An analysis of three states' alignment between language arts and mathematics standards and alternate assessments.

An analysis of three states' alignment between language arts and mathematics standards and alternate assessments. In the past, students with moderate or severe disabilities wereoften exempted from the large-scale assessments that were a keycomponent of school reform. The passage of the 1997 amendments to theIndividuals with Disabilities Education Act This article or section is currently being developed or reviewed.Some statements may be disputed, incorrect, , biased or otherwise objectionable. (IDEA), however, mandatedinclusion of all students with disabilities in accountability systems. IDEA 1997 also required that all students have access to thegeneral curriculum. At that same time, the U.S. Department ofEducation's Title I Guidance on Standards, Assessments, andAccountability (1997) emphasized that assessments for students withdisabilities, referred to as alternate assessments, should relate to thesame standards used for all students: "It is important thatstandards for students with disabilities be included in theseassessments because they are expected to meet the same standards asother students" (II. Assessments' Question #42, p. 10 of 16,as cited in Thompson Thompson, city, CanadaThompson,city (1991 pop. 14,977), central Man., Canada, on the Burntwood River. A mining town, it developed after large nickel deposits were discovered in the area in 1956. , Quenemoen, Thurlow, & Ysseldyke, 2001, p. 21).Even more recently, the No Child Left Behind Act The No Child Left Behind Act of 2001 (Public Law 107-110), commonly known as NCLB (IPA: /ˈnɪkəlbiː/), is a United States federal law that was passed in the House of Representatives on May 23, 2001 (NCLB NCLB No Child Left Behind (US education initiative), 2001) createdreporting requirements in math, language arts language artspl.n.The subjects, including reading, spelling, and composition, aimed at developing reading and writing skills, usually taught in elementary and secondary school. , and science for allstudents, including those with significant disabilities. Alternateassessments based on alternate achievement standards must be alignedwith a state's academic content standards, promote access to thegeneral curriculum, and reflect professional judgment of the highestachievement standards possible (Title I--Improving the AcademicAchievement of the Disadvantaged This article or section may contain original research or unverified claims.Please help Wikipedia by adding references. See the for details.This article has been tagged since September 2007. , 2003, 200.1(d)). Methods for determining the degree of alignment between assessmentsand state content standards have become a priority since the passage ofNCLB. Many methods of alignment are available and range from low to highcomplexity. An example of a low-complexity method is asking contentexperts to examine assessment items and match them to content standardsusing a Likert scale Likert scaleA subjective scoring system that allows a person being surveyed to quantify likes and preferences on a 5-point scale, with 1 being the least important, relevant, interesting, most ho-hum, or other, and 5 being most excellent, yeehah important, etc . Moderate-complexity alignment methods not onlyexamine the alignment of an item to a standard, but also examine theitem for another dimension, such as level of cognitive demand. Morecomplex methods examine the alignment plus many other criteria, such asbreadth of knowledge, balance of representation, and congruence con��gru��ence?n.1. a. Agreement, harmony, conformity, or correspondence.b. An instance of this: "What an extraordinary congruence of genius and era" betweenemphasis in standard and number of items used on the assessment. Bhola,Impara, and Buckendahl (2003) provide a detailed review of thesemethods. Previous research on the alignment of alternate assessments toacademic content standards used low-complexity methods to assess thedegree of alignment (Browder, Flowers et al., 2004; Browder, Spooner Spooner is an English surname of Anglo-Saxon origin, and may represent people as well as certain places : PeopleArthur Spooner, fictional character played by Jerry Stiller on The King of Queens television show ,Ahlgrim-Delzeil et al., 2003). In Browder, Flowers et al., 31states' performance indicators (i.e., the most detailed statementof state expectations for students with severe cognitive disabilities)were aligned to national reading and math content standards. Thefindings indicated that some state expectations had strong degree ofalignment to reading and math, some state expectations had weakalignment, and other states had a combination of both weak and strongdegrees of alignment. In another research study, state performanceindicators were classified as functional, academic, social, or earlychildhood (Browder, Spooner, Ahlgrim-Delzell et al.). The findingssuggested that states that had exemplary alternate assessments hadsignificantly more academic content reflected in their performanceindicators. These earlier alignment studies had two limitations. First, theresearch focus was on the alignment between state alternate assessmentperformance indicators and national academic standards, and notalternate assessment items and state academic content standards. Second,the experts were instructed to make a holistic HolisticA practice of medicine that focuses on the whole patient, and addresses the social, emotional, and spiritual needs of a patient as well as their physical treatment.Mentioned in: Aromatherapy, Stress Reduction, Traditional Chinese Medicine judgment of alignmentrather than to evaluate specific numeric numericsee numerical.numeric clustersee ten-key pad. criteria. Given the importanceof improving the technical quality of alternate assessments forinclusion in school and district adequate yearly progress Adequate Yearly Progress, or AYP, is a measurement defined by the United States federal No Child Left Behind Act that allows the U.S. Department of Education to determine how every public school and school district in the country is performing academically. (AYP AYP Adequate Yearly Progress (National Assessment of Educational Progress)AYP Anarchist Yellow PagesAYP American Youth Philharmonic )calculations under NCLB, the need exists for research that provides amore comprehensive view of the alignment between state alternateassessments and state academic content standards. This study applied and evaluated a high-complexity alignment methodto measure the degree of alignment of alternate assessment to stateacademic content standards. Three states that used performance-based andportfolio assessment formats were selected to participate in this study.The results of this study inform educators of how well alternateassessments capture language arts and mathematics constructs. METHOD An approach recommended by Webb (1997) was used to examine thealignment of alternate assessments to state general educationexpectations. The procedure combines qualitative expert judgments andquantified coding for evaluating the alignment of standards andassessments. The product of the analysis is a set of statistics thatdescribes the degree of intersection intersection/in��ter��sec��tion/ (-sek��shun) a site at which one structure crosses another. intersectiona site at which one structure crosses another. , or alignment, between the contentembedded Inserted into. See embedded system. in state content Standards and the content in stateassessments. The procedure was used to examine alternate assessmentsfrom three states in two academic areas. SELECTION OF STATE ALTERNATE ASSESSMENT SYSTEMS The first step in this research identified three states that hadexemplary alternate assessments with a clear focus on academics.Exemplary alternate assessments were selected to ensure that reviewerscould successfully align align (līn),v to move the teeth into their proper positions to conform to the line of occlusion. at least some of the general educationstandards to the assessments, and to provide practitioners with areference for evaluating degree of alignment for other alternateassessments. To identify these states, four researchers in the area ofalternate assessments were sent a questionnaire that asked them to (a)list five states with the best alternate assessments and (b) list fivestates with the strongest link between the alternate assessment andgeneral education curriculum. Three states were identified that wereconsistent across the researchers. A state department of educationrepresentative for each state agreed to participate in this study; eachrepresentative selected one typical alternate assessment for bothlanguage arts and mathematics. THE THREE STATES' ALTERNATE A SSESSMENTS State A used a performance-based alternate assessment. The readingand writing performance assessments included attending to a story andanswering questions about the content, gaining information from avariety of other sources, and creating a written product with a specificpurpose. The math performance assessment included demonstrating numbersense and computational Having to do with calculations. Something that is "highly computational" requires a large number of calculations. skills using physical models; applying simplecalculation strategies to basic addition problems; recognizing andcreating patterns using simple geometric shapes This is a list of geometric shapes. Generally composed of straight line segmentspolygon concave polygon constructible polygon ; and demonstrating anunderstanding of data collection, data display, and estimation estimationIn mathematics, use of a function or formula to derive a solution or make a prediction. Unlike approximation, it has precise connotations. In statistics, for example, it connotes the careful selection and testing of a function called an estimator. . Althoughthe tasks were the same for all students, the presentation of the tasksand response styles of the students were individualized in��di��vid��u��al��ize?tr.v. in��di��vid��u��al��ized, in��di��vid��u��al��iz��ing, in��di��vid��u��al��iz��es1. To give individuality to.2. To consider or treat individually; particularize.3. . This studyincluded State A's fourth-grade language arts and fifth-grademathematics assessments. State B's alternate assessment was a portfolio of evidencecollected over the course of an academic year of an individualstudent's performance and achievement on the established StateAcademic Expectations. Each portfolio included five entries from the sixpossible content areas (e.g., math, language arts, science) dependingupon the grade level of the student. The portfolio also included lettersfrom teachers and parents, daily schedule, and student mode ofcommunication. Teachers selected the skills to include on thestudent's alternate assessment based on the student'sindividualized education program In the United States an Individualized Education Program, commonly referred to as an IEP, is mandated by the Individuals with Disabilities Education Act (IDEA). In Canada an equivalent document is called an Individual Education Plan. (IEP IEPIn currencies, this is the abbreviation for the Irish Punt.Notes:The currency market, also known as the Foreign Exchange market, is the largest financial market in the world, with a daily average volume of over US $1 trillion. ) objectives and goals; the teacherthen aligned the skills to the state's Academic Expectations usingthe Program of Studies, which specifies grade-level performanceexpectations. This study included State B's fourth-grade languagearts and mathematics assessments. State C's alternate assessment was a portfolio of anindividual student's performance and achievement on the learningstandards Learning Standards is a term used to describe standards applied to education content, particularly in the US K-12 space.The Learning Standards themselves can can be found on the individual web sites for states [1] outlined in the Curriculum Frameworks, collected over thecourse of an academic year. Teachers selected the learning standards toaddress in an individual's portfolio. At least three pieces ofdistinct work products were required to document one or more learningstandards in the strand Strand,street in London, England, roughly parallel with the Thames River, running from the Temple to Trafalgar Square. It is a street of law courts, hotels, theaters, and office buildings and is the main artery between the City and the West End. 1. . These work products could be work samples,instructional data, and videos, audiotapes, or photographs. Portfolioscould include secondary evidence such as self-reflection self-re��flec��tionn.Self-examination; introspection.self-re��flec and letters ofsupport from others to supplement primary evidence. This study includedState C's seventh-grade language arts and sixth-grade mathematicsassessments. ALIGNMENT PROCEDURE The focus of the alignment procedure is on academic content only,and not the other dimensions Other Dimensions is a collection of stories by author Clark Ashton Smith. It was released in 1970 and was the author's sixth collection of stories published by Arkham House. It was released in an edition of 3,144 copies. (e.g., generalization gen��er��al��i��za��tionn.1. The act or an instance of generalizing.2. A principle, a statement, or an idea having general application. , self-determination self-determinationProcess by which a group of people, usually possessing a degree of political consciousness, form their own state and government. The idea evolved as a byproduct of nationalism. )that are assessed by some of the alternate assessments. The intent wasto examine the degree that alternate assessments are accessing thegeneral curriculum; therefore, each state's general curriculumstandards were used as the expectation for students with severedisabilities. Because the nomenclature nomenclature/no��men��cla��ture/ (no��men-kla?cher) a classified system of names, as of anatomical structures, organisms, etc.binomial nomenclature to describe standards and assessments isdifferent across the states, we used common language to describe thelevels of specificity within the standards. The following levels, fromthe most general statement to the most detailed description of thestandards, were used in this study: (a) subject area (e.g.,mathematics), (b) content standards (e.g., students develop number senseand use numbers and number relationships in problem-solving situationsand communicate the reasoning used in solving these problems), (c)objectives (e.g., use numbers to count, to measure, to label, and toindicate location) and (d) performance indicators (e.g., describenumbers by their characteristic, such as even, odd, prime, square). Inthis study, we used the term assessment item to represent theperformance response that could be a behavioral behavioralpertaining to behavior.behavioral disorderssee vice.behavioral seizuresee psychomotor seizure. event or a student worksample. The first step in the alignment procedure was to identify thecriteria. Content focus of the three states' alternate assessmentswas examined using four criteria recommended by Webb (1997): (a)categorical That which is unqualified or unconditional.A categorical imperative is a rule, command, or moral obligation that is absolutely and universally binding.Categorical is also used to describe programs limited to or designed for certain classes of people. concurrence CONCURRENCE, French law. The equality of rights, or privilege which several persons-have over the same thing; as, for example, the right which two judgment creditors, Whose judgments were rendered at the same time, have to be paid out of the proceeds of real estate bound by them. Dict. de Jur. h.t. , (b) range-of-knowledge correspondence, (c)balance of representation, and (d) depth-of-knowledge consistency. Categorical concurrence is the consistency of categories forcontent in the standards and assessments. In this study, we used theterm hit to indicate a content standard that has been aligned to anassessment item. Although categorical concurrence is the most obviouscriteria, additional criteria are needed to determine if the academicconstruct is being fully assessed. For example, all the assessment itemscould be aligned to only a few of the many academic content standards.Examining the range of standards an assessment covers and the balance ofassessment items across the standards provides critical informationabout how well the assessment is capturing the standards. Range-of-knowledge correspondence examines the alignment ofassessment items to the multiple objectives within the contentstandards. The range-of-knowledge numeric value is the percentage ofcontent standards with at least 50% of the objectives having one or morehits. The balance of representation criterion is used to indicate theextent to which items are evenly distributed across content standards.The formula used to compute To perform mathematical operations or general computer processing. For an explanation of "The 3 C's," or how the computer processes data, see computer. the balance of representation index is Balance = 1 -([k.summation summationn. the final argument of an attorney at the close of a trial in which he/she attempts to convince the judge and/or jury of the virtues of the client's case. (See: closing argument) over i=1][absolute value of 1/O -[I.sub.k]/H])/2, where O is the total number of objectives hit (i.e., item has beenjudged to be aligned) for the standard, [I.sub.k] is the number of itemshit corresponding to objective k, and H is the total number of items hitfor the content standard. The balance index can range from 0 (indicatesunbalanced representation) to 1.0 (indicates balance representation)with values between .6 to .7 considered a weak acceptable balance andvalues .7 or greater considered acceptable. Depth-of-knowledge (DOK DOK Daughters Of the KingDoK Disk on KeyDOK Donetsk, Ukraine - Donetsk (Airport Code)DOK Disk of Knowledge ) consistency evaluates the cognitivedemands of the standards against the cognitive demands of the alternateassessments. Completely aligned standards and assessments requires anassessment system designed to measure in some way the full range ofcognitive complexity within each specified content standard. Anacceptable level for DOK, usually rated using a 4-point scale (seeTables 1 and 2), is directly related to what is considered passing workon the assessment scale for that standard. For a more detaileddescription of the alignment criteria see Webb (1999). For the next step, we collected each of the participatingstate's standards and assessments. We conducted interviews witheach state department of education representative responsible foralternate assessment to ensure that the research team accuratelyinterpreted the documents. One outcome of these interviews was minorrewording re��word?tr.v. re��word��ed, re��word��ing, re��words1. a. To change the wording of.b. To state or express again in different words.2. of some of the states' standards, for clarity. In step three, the research team developed a description of DOKlevels for both language arts and mathematics and two coding matrices,one describing the state standards and the other listing the assessmentitems, for collecting reviewers' data. Tables 1 and 2 describe thefour levels of DOK for language arts and mathematics. The first codingmatrix listed the state's subject area, content standards, andcontent objectives. Reviewers used this coding matrix to evaluate DOKfor each state's general education content objective. The statestandards coding matrix served two purposes: the results of DOK codingfor each objective provided evidence of the DOK expectations for eachstate, and the reviewers became familiar with each of the states'standards. The second coding matrix listed each item/activity on theassessment with columns for reviewers to code (a) DOK, (b) primarycontent standard hit, and (c) secondary content standard hit. In step 4, reviewers were recruited and trained to align theassessments to the standards. To provide a diverse perspective of thealternate assessments, reviewers from both general and special educationbackgrounds were asked to evaluate the alignment of the state standardsand alternate assessments. Three different groups were used asreviewers: (a) six content experts with experience in test developmentand standards writing (three language arts and three mathematics), (b)three state directors of alternate assessment, and (c) four researcherswho included two national experts in severe disabilities. All reviewers attended a 1-day training session. The first part ofthe training involved describing the three states' alternateassessments. Copies of the alternate assessments were available for eachreviewer re��view��er?n.One who reviews, especially one who writes critical reviews, as for a newspaper or magazine.reviewerNouna person who writes reviews of books, films, etc.Noun 1. to examine. Next, the reviewers read the definitions of DOKlevels and were asked to think about typical behaviors for the gradelevel. Language arts and math experts were present to answer specificquestions and provide detailed examples for reviewers. Then thereviewers were provided five performance indicators from a statestandard and, as a group, discussed the DOK for each performanceindicator. They discussed the ratings and reached a consensus beforecontinuing to the next performance indicator. The reviewers thenindependently coded DOK for a sample of five alternate assessment items,compared their ratings, and discussed areas of agreement anddisagreement. Any disagreements were discussed until consensus wasreached with the facilitation FacilitationThe process of providing a market for a security. Normally, this refers to bids and offers made for large blocks of securities, such as those traded by institutions. of the general education content expert inthe respective subject area. It should be noted that it was notnecessary to get exact agreement among reviewers because results acrossreviewers are averaged in the DOK. Next, we instructed the reviewers to link 10 assessment items tothe state standards. Five of the items were linked as a group withdiscussion led by an expert in language arts and mathematics, and fiveitems were rated independently. Reviewers' links were discussed andconsensus was reached before discontinuing the activity. The final step was independent reviewer coding of the alignment ofassessments to standards. All reviewers were provided a description ofthe assessment system they were responsible for evaluating, DOKdescriptions for the appropriate subject domain, and the two codingmatrices. Instructions for coding were similar to those used in thetraining except all coding was done independently. RESULTS Table 3 shows the type of alternate assessments, number ofreviewers, and number and percentage of items that aligned to astandard. The number of items on the assessments ranged from 5 to 54.The mean number of items that hit a standard ranged from 4 to 51.8; thepercentage of items that could be aligned to a content standard rangedfrom 77.5% to 94.1%. The performance-based alternate assessments tendedto have a higher percentage of items that were aligned to the standardsthan the portfolio assessments. CATEGORICAL CONCURRENCE Table 4 reports categorical concurrence consistencies for allstates. According to according toprep.1. As stated or indicated by; on the authority of: according to historians.2. In keeping with: according to instructions.3. Webb (1999), an acceptable standard for categoricalconcurrence consistencies is achieved if all the content standards haveat least six hits. In this study none of the states achieved 100%categorical concurrence; in other words Adv. 1. in other words - otherwise stated; "in other words, we are broke"put differently , none of the states had at leastsix hits in all the content standards. State A, the performance-basedassessments, had the highest level of categorical concurrences for bothlanguage arts (50%) and mathematics (66.7%); for language arts, three ofthe six content standards had at least six hits and for mathematics,four of the six content standards had six hits. These assessments alsohad the greatest number of assessment items, providing a greateropportunity for hits; conversely con��verse?1?intr.v. con��versed, con��vers��ing, con��vers��es1. To engage in a spoken exchange of thoughts, ideas, or feelings; talk. See Synonyms at speak.2. , assessments with the lowest number ofitems tended to have the lowest level of categorical concurrence. StateC mathematics assessment, which had only five items, had the lowestlevel of categorical concurrence (0%). The trend in the data suggeststhat there is a direct relationship between the number of items on theassessment and categorical concurrence. RANGE-OF-KNOWLEDGE CORRESPONDENCE Range-of-knowledge correspondence reflects the Consistency of thebreadth of knowledge required in both the standards and the assessment.The range-of-knowledge value is the percentage of content standards withat least 50% of the objectives having One or more hits. The number andpercentage of objectives with at least one hit for all slates arereported in Table 5. The range-of-knowledge was not calculated for StateB because they have a single-level set of standards; they have contentstandards but do not divide the content standards into objectives. StateA had a range-of-knowledge congruency con��gru��en��cy?n. pl. con��gru��en��ciesCongruence. of 36.5% in language arts and22.0% in mathematics. State C language arts and math had a 0%range-of-knowledge congruency. Again, the number of items on theassessment, as well as the range of items spread across the standards,had a direct relationship to the number of objectives that could be hitwithin a content standard. BALANCE OF REPRESENTATION The balance of representation examines the distribution ofassessment items across the objectives. In this study, the assumption isthat items should be evenly spread among the content standards. Table 6presents a summary of balance indices for all the states. State Alanguage arts had an acceptable balance index (i.e., greater than .70)for two of the six content standards, resulting in a 33.3% acceptablebalance-of-knowledge consistency. None of the other state assessmentshad a balance index greater than .6. This would indicate that most ofthe assessments focused on a small number of objectives within thecontent standards. DEPTH-OF-KNOWLEDGE (DOK) CONSISTENCY DOK consistency reflects the extent to which the assessment itemsmeasure the range of expected knowledge. Before examining DOK, thereliability (i.e., coefficient coefficient/co��ef��fi��cient/ (ko?ah-fish��int)1. an expression of the change or effect produced by variation in certain factors, or of the ratio between two different quantities.2. alpha) of reviewer ratings for assessmentitem DOK was estimated (Shrout & Fleiss, 1979). The average measureof intraclass correlation In statistics, the intraclass correlation (or the intraclass correlation coefficient[1]) is a measure of correlation, consistency or conformity for a data set when it has multiple groups. ranged from .69 to .97, suggesting adequateconsistency between reviewer DOK ratings. DOK results are summarized in Table 7. The DOK value represents thepercentage of content standards that have 50% of the assessment items ator above the state standards' DOK. Only State C language arts hadone of the three content standards (33.3%) where 50% of the items wereat or above the state standards' DOK levels. None of the otherassessments had any content standard with DOK levels equivalent to thestate standards. A similar pattern was consistent across allassessments; the DOK level for the state standards tended to haveratings at the higher end of the scale, and the alternate assessmentstended to have ratings at the lower end of the scale. It is of interestto note that almost all of the assessments had some items that wererated 3 (strategic thinking), and State C language arts had 39% of theassessment items rated 3 or higher. FEEDBACK FROM THE STATES' REPRESENTATIVES All three state department of education representatives reportedthat the outcomes were useful and the alignment method seemedappropriate. Much of the lack of congruence between the assessments andstate standards was due to the intentional in��ten��tion��al?adj.1. Done deliberately; intended: an intentional slight.See Synonyms at voluntary.2. Having to do with intention. focus of the alternateassessment to modified content standards. For example, States B and Crequired a limited number of content standards to be addressed on thealternate assessments; in other words, the states do not expect all thecontent standards to be addressed. Although most of the alignmentstatistics did not meet the acceptable level as prescribed pre��scribe?v. pre��scribed, pre��scrib��ing, pre��scribesv.tr.1. To set down as a rule or guide; enjoin. See Synonyms at dictate.2. To order the use of (a medicine or other treatment). by Webb, allthe state representatives wanted the higher expectations for theiralternate assessments applied to their alternate assessment. As onerepresentative stated, "We want the higher expectations, eventhough we may not get there." All three representatives also wantedthe alternate assessment to sample the entire DOK spectrum so thatstudents would not be underestimated. Although they acknowledged thatthe alternate assessments might be skewed skewedcurve of a usually unimodal distribution with one tail drawn out more than the other and the median will lie above or below the mean.skewedEpidemiology adjective Referring to an asymmetrical distribution of a population or of data toward lower DOK levels, theyconsidered it important to have some items at the highest DOK level. DISCUSSION None of the state alternate assessments met the recommended levelfor any of the alignment criteria established for general assessments.Similarly, many general education assessments, which typically have moreitems, do not meet an acceptable alignment level The alignment level in an audio signal chain or on an audio recording is a defined anchor point that represents a reasonable or typical level. It does not represent a particular sound level or signal level or digital representation, but it can be defined as corresponding to (see Webb, 1999, 2002;Webb, Horton Horton,river, c.275 mi (440 km) long, rising in a lake N of Great Bear Lake, Northwest Territories, Canada, and flowing NW to Franklin Bay, a part of the Beaufort Sea. , & O'Neal, 2002). For example, in a generaleducation alignment study of four states' language artsassessments, acceptable alignment levels were met for 75% on thecategorical concurrence, 31% for depth-of-knowledge, 47% for range ofknowledge, and 31% for balance of representation (Webb, 2002). The alignment statistics do not suggest that the alternateassessments in this study are of poor quality; alignment procedures donot describe an attribute of the assessment or standards, but rather therelationship between the standards and the assessment. In fact, theseresults indicate that there is a relationship between alternateassessments and academic standards, but the strength of thatrelationship is not as strong as Webb (2002) recommends it should be.Webb's recommendations are based on his work with general educationmathematics and science assessments; these assessments tend to have manymore items than alternate assessment and are usually given in apaper/pencil format. Given that alternate assessments were designedusing modified academic standards (i.e., extended standards or expandedbenchmarks) and alternate achievement standards, we may not findalignment statistics that meet criteria established by Webb. The overallusefulness of the information obtained, the potential need for furthercurriculum development, and the need to examine the alignment criteriaused for general education will each now be considered. POTENTIAL USEFULNESS TO STATES These results provided the states with important information abouthow their alternate assessments were addressing three critical elements:(a) academic content standards, (b) academic achievement standards, and(c) assessments aligned to those standards. Most of the alternateassessment items were aligned to the academic content standards, rangingfrom 77% to 94% of items hitting a standard. This provides evidence thatthese exemplary alternate assessments were accessing the generalcurriculum. Specifically, the alternate assessments were measuring someaspect of language arts and mathematics. Whereas the Browder, Flowers etal. (2004) alignment study found that some states' alternateassessments reflected academic content, the current study's methodprovides states with quantitative measures of the degree of thealignment. Assessment items-whether identified by the state in aperformance assessment or compiled by teachers in a portfolio--wereindividually matched to the state's general education standard bycontent area experts. This backmapping provides a clear picture of theextent to which the assessments are measuring the state standards'academic content such as language arts or measuring "somethingelse." This "something else" may be values the stateholds that extend beyond the academic content. For example, State Bestablished additional dimensions that are important for students withsevere disabilities: (a) support, (b) setting, (c) social relationships,and (d) self-determination. When focusing on outcomes that extend beyondstate academic standards, it may be important to articulate articulate/ar��tic��u��late/ (ahr-tik��u-lat)1. to pronounce clearly and distinctly.2. to make speech sounds by manipulation of the vocal organs.3. to express in coherent verbal form.4. theseoutcomes as well and determine if the assessments align to theseoutcomes. An alternate achievement standard is an expectation of performancethat differs in complexity from a grade-level achievement standard.States are permitted to use alternate achievement standards to evaluatethe performance of students with severe cognitive disabilities. The DOKoutcomes were useful in helping states examine the range of cognitivecomplexity of assessment items across state standards. Althoughalternate achievement standards are allowed, alternate assessmentsshould reflect the highest achievement standards possible. All the staterepresentatives wanted assessment items that assessed higher levels ofcognitive complexity. A study of alignment should examine item format and other ways thestructure of knowledge can be represented in alternate assessments(Webb, 1999). The results suggest that alternate assessmentformats--performance-based or portfolio assessments--are relevant to thealignment procedures. The portfolio assessments had a lower number ofitems than the performance-based assessments; this makes it difficultfor portfolio assessments to attain an acceptable level for thecategorical concurrence and the range-of-knowledge criteria usingWebb's criteria. Consideration of the different components of alignment can help astate develop a comprehensive picture of how well the alternateassessment aligns to state standards. It should be noted that thecurrent study focused on a single grade level in each academic area. Astate application of this alignment method may need to consideralternate assessments at multiple grade levels to get a full picture ofhow alternate assessments are accessing the general curriculum. NEED FOR FURTHER CURRICULUM DEVELOPMENT One of the challenges in developing alternate assessments is thatthe range of examples in research for teaching academics to studentswith severe disabilities, and textbooks on this population, do notextend much beyond sight word training in reading and money skills inmath (Browder, Spooner, Ahlgrim-Delzell et al., 2003). States thatidentify performance items and teachers who document portfolios need tobe creative when developing assessment items in the broader componentsof language arts and math. One possible response to the results found inthe current study would not be to alter Webb's (1997) criteria forgeneral education assessments when evaluating alternate assessments, butinstead to consider how these outcomes inform us about the need forongoing curriculum development. Another interesting outcome is that all of the states had somealternate assessment items at the upper DOK levels (see Table 7). Allthree state representatives thought that the inclusion of these morecognitive demanding items, which required students to go beyond simpleapplications and use reasoning and evaluation skills, were essential forstudents with significant cognitive dis abilities as well. A carefulreview of the upper level skills found in these assessments may generatenew ideas for enabling students with complex and multiple disabilitiesto demonstrate these skills in meaningful ways. DEGREES OF ALIGNMENT FOR ALTERNATE A SSESSMENTS Another issue is the need to consider acceptable degrees ofalignment for alternate assessments. For example, none of the alternateassessments met the range-of-knowledge criterion. In contrast, most ofthe states narrow and prioritize pri��or��i��tize?v. pri��or��i��tized, pri��or��i��tiz��ing, pri��or��i��tiz��es Usage Problemv.tr.To arrange or deal with in order of importance.v.intr. within the general curriculum indeveloping their alternate assessments, so it would not be expected thatalternate assessment would be aligned across all the states'general education curriculum standards. Most of the states focused thealternate assessments on two standards. If we had evaluated therange-of-knowledge criterion based on the narrowed standards, most ofthe state alternate assessments would have met this criterion; that is,they would have had at least one item aligned to a target contentstandard. Thus, if one way states define alternate achievement standardsis for the student to demonstrate fewer standards (e.g., two standards)within a content domain (e.g., math), it would be appropriate todetermine if this subset A group of commands or functions that do not include all the capabilities of the original specification. Software or hardware components designed for the subset will also work with the original. were reflected in the assessment items. Similarly, the DOK for alternate assessment items was evaluatedbased on the DOK for the general education curriculum. Although none ofthe states met the criterion, most of the states had items with a DOK of3 (i.e., involved multiple steps to correctly answer the item).Alternate assessments were skewed toward items at the lower end of DOK.Like limiting the number of standards to be assessed, having more itemsat entry levels of DOK may be appropriate for students with significantcognitive disabilities, but including assessment items that arecognitively demanding provides an opportunity for students todemonstrate knowledge at the highest achievement standards possible. Oneoption, used by State C, is to incorporate the level of complexity ofthe item/activity into the scoring of the alternate assessment. Thisprovides an incentive for teachers to challenge students with severedisabilities with more complex activities, while also communicatingwhich set of alternate achievement standards are applied. RECOMMENDATIONS FOR PRACTICE AND FUTURE RESEARCH Based on this research, we offer the following recommendations foraligning alternate assessments to state expectations. First, statesshould document how they align the alternate assessment items to thegeneral education curriculum standards. This allows a state to evaluatehow well its alternate assessments are accessing the general curriculum.Documentation will also establish whether in modifying the generaleducation curriculum standards to be used with alternate assessments,the original intent of the academic standards was maintained. In ourearlier work (Browder, Flowers et al., 2004), we found that some stateshad weak links between their alternate assessment items and academiccontent. This may occur if curriculum content experts are not involvedin developing the alternate assessments and if the"extensions" of state standards used for developing alternateassessments use too liberal definitions of the academic components. Thestates chosen for the current study were all ones that carefullypreserved the focus on their state academic content standards, with theresult that nearly all assessment items could be matched back to statestandards in language arts and math. The second recommendation is that states should articulate howtheir academic content standards for alternate assessments will affectalignment, prior to conducting the alignment review. If statesintentionally in��ten��tion��al?adj.1. Done deliberately; intended: an intentional slight.See Synonyms at voluntary.2. Having to do with intention. address only some of the standards and weight theimportance of them, these narrowed and weighted criteria should be usedto examine range of knowledge and balance of representation. It stillmay be of interest for states to examine what other parts of the generaleducation curriculum are being emphasized in the alternate assessmentsthrough a broader review, but the goal would be that the narrowed listof standards are dearly represented. Third, states are encouraged to document the depth of knowledge intheir alternate assessments. Although it may be inappropriate to expectstudents with severe disabilities to function at the same expected DOKas general education students, all students should have the opportunityto demonstrate their highest DOK. CONCLUSIONS Alternate assessments are changing rapidly, and our understandingof what they can and should be is changing as well. It is important tonote that the states in this study continue to evaluate and revise theiralternate assessments. The degree of alignment of alternate assessmentsand expectations may not reflect the current alignment study of thesestates. In particular, new guidelines guidelines,n.pl a set of standards, criteria, or specifications to be used or followed in the performance of certain tasks. on alternate achievement standardsmay provide incentive to make modifications in alternate assessments.States will need to continue to develop alternate assessments to addressalternate achievement standards by grade level. Although this study focused on academic content standards, someeducators may disagree with Verb 1. disagree with - not be very easily digestible; "Spicy food disagrees with some people"hurt - give trouble or pain to; "This exercise will hurt your back" developing a curriculum based on academiccontent standards. Agran, Alper, and Wehmeyer (2002) found that teachersdid not consider access to the general curriculum important for studentswith severe disabilities, but instead ranked functional and socialskills as most important reflecting the curricular loci loci[L.] plural of locus.lociPlural of locus, see there of the last 2decades. Although teacher perceptions may have changed in the yearssince this survey was conducted, the strong emphasis on academics forgedby alternate assessment has curricular implications for students withsevere disabilities that merit further discussion. For example, inresponding to the NCLB requirement to assess and teach all studentsreading, math, and science, how do educators preserve other curricularpriorities as well? In summary, this study applied a systematic alignment procedure forevaluating the match between alternate assessments and state academicstandards. By considering not only whether each item aligns to academiccontent standards, but also the depth of knowledge and breadth ofstandards addressed, a more complete picture of how alternateassessments are accessing the general curriculum can be assessed. Inthis research, the outcomes reflect the need for experts in severedisabilities and academic content to determine alignment criteria andacceptable alignment levels that promote access to the generalcurriculum and reflect the highest standards possible for students withsevere cognitive disabilities. REFERENCES Agran, M., Alper, S., & Wehmeyer, M. L. (2002). Access to thegeneral curriculum for students with significant disabilities: What itmeans for teachers. Education and Training in Mental Retardation mental retardation,below average level of intellectual functioning, usually defined by an IQ of below 70 to 75, combined with limitations in the skills necessary for daily living. andDevelopmental Disabilities, 37, 123-133. Bhola, D. S., Impara, J. C., & Buckendahl, C. W. (2003).Aligning tests with states' content standards: Methods and issues.Educational Measurement: Issues and Practice, Fall, 21-29. Browder, D. M., Flowers, C., Ahlgrim-Delzell, L., Karvonen, M.,Spooner, E, & Algozzine, R. (2004). The alignment of alternateassessment content to academic and functional curricula. Journal ofSpecial Education, 37, 211-223. Browder, D. M., Spooner, E, Ahlgrim-Delzell, L., Flowers, C.,Algozzine, tL, & Karvonen, M. (2003). A content analysis of thecurricular philosophies reflected in states' alternate assessments.Research and Practice for Persons With Severe Disabilities, 2, 165-181. Browder, D. M., Spooner, E, Algozzine, R. AhlgrimDelzell, L.,Flowers, C., & Karvonen, M. (2003). What we know and what we need toknow about alternate assessment. Exceptional Children, 70, 45-61.Individuals with Disabilities Education Act of 1997, 120 U.S.C.[section]1400 et seq et seq.(et seek) n. abbreviation for the Latin phrase et sequentes meaning "and the following." It is commonly used by lawyers to include numbered lists, pages or sections after the first number is stated, as in "the rules of the road are found in Vehicle Code . No Child Left Behind Act of 2001, Pub. L. No. 107-110, 115 Stat.1425 (2001). Shrout, P. E. & Fleiss, J. L. (1979). Intraclass correlations:Uses in assessing rater rat��er?n.1. One that rates, especially one that establishes a rating.2. One having an indicated rank or rating. Often used in combination: a third-rater; a first-rater.reliability. Psychological Bulletin, 86(2)420-428. Thompson, S. J., Quenemoen, R. E, Thurlow, M. L., & Ysseldyke,J. E. (2001). Alternate assessments for students with disabilities.Thousand Oaks Thousand Oaks,residential city (1990 pop. 104,352), Ventura co., S Calif., in a farm area; inc. 1964. Avocados, citrus, vegetables, strawberries, and nursery products are grown. , CA: Corwin Corwin may refer to:People: Chris Corwin, a business owner in Iowa City David Corwin, a psychiatrist in Colorado Franklin Corwin, a U.S. Representative from Illinois Jeff Corwin, a television show host on Animal Planet Press. Title I--Improving the Academic Achievement of the Disadvantaged;Final Rule, 68 Fed. Reg REG,n.pr See random event generator. . 236 (Dec. 9, 2003). U.S. Department of Education. Title I Guidance on Standards,Assessments, and Accountability (1997). Washington, DC: Author. Webb, N. L. (1997). Criteria for alignment of expectation andassessments in mathematics and science education. (NISE NISE National Institute for Science EducationNISE Nanoscale Informal Science Education (Network)NISE NCCOSC In-Service EngineeringNISE Naval In-Service EngineeringNISE Network Installation Service Engineer ResearchMonograph No. 6). Madison Madison, cities, United StatesMadison.1 City (1990 pop. 12,006), seat of Jefferson co., SE Ind., on the Ohio River; settled c.1806, inc. 1838. It is a port of entry and a tobacco marketing center. : University of Wisconsin-Madison “University of Wisconsin” redirects here. For other uses, see University of Wisconsin (disambiguation).A public, land-grant institution, UW-Madison offers a wide spectrum of liberal arts studies, professional programs, and student activities. , NationalInstitute for Science Education. Washington, DC: Council of Chief StateSchool Officers The Council of Chief State School Officers (CCSSO) is a national nonprofit organization in the United States which represents public officials that head elementary and secondary education departments. . Webb, N. L. (1999). Alignment of science and mathematics standardsand assessments in four states. (NISE Research Monograph No. 18).Madison: University of Wisconsin-Madison, National Institute for ScienceEducation. Washington, DC: Council of Chief State School Officers. Webb, N. L. (2002). Alignment study in language arts, mathematics,Science, and social studies of state standards and assessments for fourstates. Washington, DC: Council of Chief State School Officers. Webb, N. L., Horton, M., & O'Neal, S. (2002, April). Ananalysis of the alignment between language arts standards andassessments for four states. Paper presented at the meeting of theAmerican Educational Research Association, New Orleans New Orleans(ôr`lēənz –lənz, ôrlēnz`), city (2006 pop. 187,525), coextensive with Orleans parish, SE La., between the Mississippi River and Lake Pontchartrain, 107 mi (172 km) by water from the river mouth; founded , LA. CLAUDIA Claudia(klôd`ēə), Christian who sent greetings to Timothy, as recorded in Paul's Letter to Timothy. Claudiaproves innocence by rescuing goddess’ ship. [Rom. Myth.: Hall, 70]See : Chastity FLOWERS, Associate Professor, Department of EducationalLeadership, Research, and Technology; DIANE BROWDER (CEC (Central Electronic Complex) The set of hardware that defines a mainframe, which includes the CPU(s), memory, channels, controllers and power supplies included in the box. Some CECs, such as IBM's Multiprise 2000 and 3000, include data storage devices as well. North Carolina North Carolina,state in the SE United States. It is bordered by the Atlantic Ocean (E), South Carolina and Georgia (S), Tennessee (W), and Virginia (N).Facts and FiguresArea, 52,586 sq mi (136,198 sq km). Pop. Council of Exceptional Children), Snyder Distinguished Professor,Department of Special Education and Child Development; and LYNNAHLGRIM-DELZELL, Research Associate, University of North Carolina atCharlotte. Correspondence concerning this article should be addressed toClaudia Flowers, Department of Educational Administration, Research, andTechnology, University of North Carolina at Charlotte, 9201 UniversityCity Blvd., Charlotte, NC 28223. (email: cpflower@email.uncc.edu) Support for this research was funded in part by Grant No.H324C010040 of the U.S. Department of Education, Office of SpecialEducation Programs, awarded to the University of North Carolina atCharlotte. The opinions expressed do not necessarily reflect theposition or policy of the Department of Education, and no officialendorsement should be inferred. Manuscript manuscript,a handwritten work as distinguished from printing. The oldest manuscripts, those found in Egyptian tombs, were written on papyrus; the earliest dates from c.3500 B.C. received July 2004; accepted November 2004.TABLE 1Language Arts Description of Depth-of-Knowledge LevelsLevel 1 Level 2Requires students to Requires processingrecall, observe, question, beyond recall andor represent facts or observation. Requiressimple skills or abilities. both comprehension andRequires only surface subsequent processing ofunderstanding of text, text. Involves ordering,often verbatim recall. classifying text as well as identifying patterns,Level 1 items involve relationships, and mainliteral, explicit, "right points.there" thinking, one-step responses. Level 3 Level 4Requires students to go Requires extended higherbeyond text. Requires order processing. Typicallystudents to reason, requires extended timegeneralize, connect ideas, (more than 10 minutes) toand develop a plan or complete task, but timesequence of steps. Usually spent not on repetitivetakes less than 10 minutes tasks. Involves takingto complete task. Involves information from oneinferences, prediction, text/passage and applyingelaboration, and summary. this information to a newRequires students to task. May requiresupport positions using generating hypothesesprior knowledge and to and performing complexmanipulate themes across analyses and connectionspassages. There may be among texts.more than one possibleanswer. Level 4 items involve application andLevel 3 items involve integration with back-application, inferences, ground knowledge;and reading between the there are multiplelines. answers and the answer lies within the individual.Note. Reviewer coded "0" for prerequisite skill.TABLE 2Mathematics Description of Depth-of-Knowledge LevelsLevel 1 Level 2Requires students to Requires students to makerecall or observe facts, decisions on how todefinitions, terms. approach a problem.Involves simple one-step Requires students toprocedures. Involves compare, classify, organize,computing simple estimate, or order data.algorithms (e.g., sum, Typically involves two-quotient). step procedures. Level 2 items include word problems with simple one- step solutions, graphing and regrouping. Level 3 Level 4Requires reasoning, Requires complexplanning, or use of reasoning, planning,evidence to solve problem developing and thinking.or algorithm. May involve Typically requiresactivity with more than extended time to completeone possible answer. problem, but time spentRequires conjecture or not on repetitive tasks.restructuring of problems. Requires students toInvolves drawing make several connectionsconclusions from and apply one approachobservations, citing among many to solve theevidence, and developing problem. Involves complexlogical arguments for restructuring of data,concepts. Uses concepts establishing andto solve nonroutine evaluating criteria toproblems. solve problems.Level 3 items include Level 4 items are project-open-ended word based, involveproblems where the explanation &operation is not given. justification.Note. Reviewer coded "0" for prerequisite skill.TABLE 3Description of Alternate Assessment, Number of Reviewers, andMean Number of Assessment Items With a Hit Type Subject Grade Assessment Number ofState Assessment Domain Level Items Reviewers N NA Performance- LA 4 33 7 based Math 5 54 8B Portfolio LA 4 28 7 Math 4 26 8C Portfolio LA 7 21 9 Math 6 5 10 Assessment Type Subject Items ThatState Assessment Domain Hit Standard M %A Performance- LA 30.3 91.8 based Math 51.8 94.1B Portfolio LA 21.7 77.5 Math 20.5 78.8C Portfolio LA 18.4 92.4 Math 4.0 80.0Note. LA = language arts.TABLE 4Categorical Concurrence Criterion for Three States in Language Artsand Mathematics Rank-Ordered Content Standard Content Assessment ContentState Area Items Standards 1st 2nd 3rd M M M N N (SD) (SD) (SD)A LA 33 6 14.4 11.1 7.7 (7.8) (11.3) (4.3) Math 54 6 32.3 17.1 15.0 (10.3) (6.7) (9.2) LA 28 5 26.1 6.9 2.0 (10.6) (3.3) (1.8) Math 26 5 22.9 6.0 2.0 (10.1) (4.4) (2.6)C LA 21 3 16.6 4.8 4.3 (4.3) (1.6) (5.1) Math 5 5 4.9 .8 .0 (2.1) (1.1) (.0) Rank-Ordered Content Standard Content NumberState Area 4th 5th 6th of Hits C[C.sup.I] M M M M (SD) (SD) (SD) (SD) %A LA 5.0 1.6 .0 39.8 50.0 (5.0) (1.5) (.0) (17.6) Math 7.0 3.9 .0 75.3 66.7 (5.3) (3.9) (.0) (15.1) LA 1.0 .4 36.4 40.0 (1.7) (.8) (9.6) Math .4 .3 31.6 40.0 (1.1) (.8) (11.3)C LA 25.7 33.3 (5.6) Math .0 .0 5.7 .0 (.0) (.0) (2.9)Note. Content standards are in rank-ordered by mean number of hitsand do not correspond to the states numbering of content standards.C[C.sup.1] is the percentage acceptable categorical concurrence.LA = language arts.TABLE 5Range-of-Knowledge (ROK) for States A and C Content Assessment ContentState Area Items (N) Standards (N) Objectives (N)A LA 33 6 28 Math 54 6 37C LA 21 3 19 Math 5 5 17 Content Standards Percentage ContentState Area 1st 2nd 3rd 4th 5th 6th ROK (1)A LA 100 80 25 14 0 0 36.5 Math 50 20 14 33 0 0 22.0C LA 33 17 43 .0 Math 33 0 0 0 0 .0Note. (1) Range-of-knowledge (ROK) correspondence is the percentage ofcontent standards with at least 50% of the objectives with one hit.LA = language arts.TABLE 6Balance of Representation Indices and Percentage of Content StandardsWith Acceptable Balance Indices Content Standards (Rank-Ordered) ContentState Area 1st 2nd 3rd 4th 5th 6thA LA 1.00 (a) .31 .79 .31 .61 NA Math .53 .28 .20 .17 .22 NAC LA .35 .40 .59 Math .43 .57 NA NA NA Content Standards (Rank-Ordered) Content % AcceptableState Area % Acceptable WeakA LA 33.3 50.0 Math .0 .0C LA .0 .0 Math .0 .0Note. (a) Only one objective for content standard one. NA hadno hits. LA = language arts.TABLE 7Depth-of-KnowledKe (DOK) Consistency Between Standards and Assessment DOK (a) Distribution for Performance IndicatorsState Content Area DOK PI (b) 1 2 3 4 N % % % %A LA 0 87 20 32 29 18 Math 0 84 <1 29 42 23B LA 0 255 15 31 37 17 Math 0 325 20 40 34 6C LA 33.3 295 22 32 31 15 Math 0 215 15 47 32 6 DOK (a) Distribution for Assessment ItemsState Content Area Items 0 1 2 3 4 N % % % % %A LA 33 5 55 25 14 1 Math 54 4 55 35 6 0B LA 28 18 22 39 19 1 Math 26 17 53 28 2 0C LA 21 11 23 27 36 3 Math 5 28 56 16 0 0Note. (a) Depth-of-knowledge (DOK) levels are (0) prerequisite skill,(1) simple recall, (2) skill or concept, (3) strategic thinking,and (4) extended thinking. (b) Performance Indicators.LA = language arts.