Standards-referenced assessment for vocational education and training in schools.

Standards-referenced assessment for vocational education and training in schools. This study examined a model of assessment that could be appliednationally for Year Twelve Vocational Education vocational education,training designed to advance individuals' general proficiency, especially in relation to their present or future occupations. The term does not normally include training for the professions. and Training (VET)subjects and which could yield both a differentiating score andrecognition of competence. More than fifty colleges across all statesand territories of Australia The states and territories of Australia make up the Commonwealth of Australia under a federal system of government.States and territories of AustraliaStatesState Abbreviation Capital field-tested the approach over one schoolyear. Results showed that the model allowed for a standards-referencedmodel to be used: that the approach was compatible with the diverserange of senior secondary assessment systems in use throughout Australia Australia(ôstrāl`yə), smallest continent, between the Indian and Pacific oceans. With the island state of Tasmania to the south, the continent makes up the Commonwealth of Australia, a federal parliamentary state (2005 est. pop. and that there were considerable cost benefits to be had in adopting thelogic of item response modelling for the development of rubrics forscoring performances on units of competence from National TrainingPackages. A change in the logic of competency COMPETENCY, evidence. The legal fitness or ability of a witness to be heard on the trial of a cause. This term is also applied to written or other evidence which may be legally given on such trial, as, depositions, letters, account-books, and the like. 2. assessment was proposed,in that the performance indicators were not rated using a dichotomy di��chot��o��my?n. pl. di��chot��o��mies1. Division into two usually contradictory parts or opinions: "the dichotomy of the one and the many"Louis Auchincloss. butwith a series of quality ordered criteria to indicate how well studentsperformed specified tasks in the workplace or its simulation. The studyvalidated val��i��date?tr.v. val��i��dat��ed, val��i��dat��ing, val��i��dates1. To declare or make legally valid.2. To mark with an indication of official sanction.3. the method of assessment development, demonstrated themethod's consistency, and showed how the method could address theissue of consistency across states. The study also proposed a set ofprinciples for a joint assessment of both quality and competence. ********** The introduction of competency-based education to Australia in 1992was intended to increase the skill level of the Australian Australianpertaining to or originating in Australia.Australian bat lyssavirus diseasesee Australian bat lyssavirus disease.Australian cattle doga medium-sized, compact working dog used for control of cattle. workforce.Its introduction was accompanied by a debate about how the competenciesshould be assessed and other issues (such as grades) that were morerelated to reporting than assessment. The discussions were hampered by alack of clarity of the terminology, shifting definitions of basicconcepts and inconsistent use of language to describe the process(Stanley Stanley,town (1991 pop. 1,557), capital of the Falkland Islands, S Atlantic Ocean, on East Falkland island. It is the main port and trading center of the islands. The name is sometimes written as Port Stanley. , 1993). The debate surrounding sur��round?tr.v. sur��round��ed, sur��round��ing, sur��rounds1. To extend on all sides of simultaneously; encircle.2. To enclose or confine on all sides so as to bar escape or outside communication.n. the nature of competency-based assessmentcentred on two issues: first, whether grading was a suitable approach touse with competency assessment; and second, whether criteria forassessment can be generic or specific. Discussion of both issues tends to have been confused by imprecise im��pre��cise?adj.Not precise.impre��cisely adv. use of terminology and appears to have ignored the considerable body ofliterature available on each topic. The lack of rigorous research andtheoretical models has been detrimental det��ri��men��tal?adj.Causing damage or harm; injurious.detri��men to the development of the field. One of the few comprehensive studies into this field (McCurry,2003) has shown that generic criteria cannot be applied effectively tocompetence assessments, and therefore that debate will not be extendedhere. Rather, this article focuses on the application of specificcriteria in the development of assessment procedures to provide adifferentiating score for students who chose to study vocationalsubjects in their last year of secondary school and who, under theregime of the dichotomy between the existence or absence of competency,were at a disadvantage to those students studying subjects graded onlevels of quality when university selection procedures were implemented. Griffin, Gillis, Keating Keating may refer refer to the following: PeopleFor people with the surname Keating, see Keating (surname) PlacesSeveral places in the US: Keating Township, Potter County, Pennsylvania Keating Township, McKean County, Pennsylvania and Fennessy (2001) examined the use ofstandards--referenced assessments in vocational education and Griffinand Gillis (2001; 2002) outlined possible procedures that could be usedto examine the efficacy of a differentiated scoring system Noun 1. scoring system - a system of classifying according to quality or merit or amountrating systemclassification system - a system for classifying things for VET inschools. Their research has shown how current approaches tocompetency-based assessment could be used to yield a differentiatedscore in addition to the recognition of competence, without altering thefundamentals of the entrenched en��trench? also in��trenchv. en��trenched, en��trench��ing, en��trench��esv.tr.1. To provide with a trench, especially for the purpose of fortifying or defending.2. competency-based approach but insteadfocusing on a customisation of record-keeping and reporting frameworks.This article reports on how those approaches were tested and the modelof assessment for competency-based education that emerged. Competency assessment and standards-referenced assessment A major purpose of competency assessment has been to provide forthe recognition of competence. This article argues that it can alsoprovide evidence of differentiation for selection purposes. Observationsused to provide for recognition or differentiation can all be based onthe procedures already used by assessors in the workplace, endorsed bythe Australian National Training Authority and national industrytraining advisory boards and documented in the relevantassessor-training package (National Assessors and Workplace TrainersBody, 1998). When competency-based assessment was introduced into vocationaleducation in Australia in the early 1990s it was presented as an exampleof criterion-referenced or criterion-based assessment, but this was alimited and misleading view of criterion referencing. Over the best partof a decade, the discussion was based on the use of a dichotomy thatrecognised only the existence or absence of competence and excluded theidea of the quality of performance. In 2001, Griffin et al. argued thatthe definition of criterion-referenced assessment could itself be usedto justify the approach of the dichotomy of competent ornot-yet-competent and yet still yield a differentiating score. Theyargued that competence was itself a decision point within a continuum Continuum (pl. -tinua or -tinuums) can refer to: Continuum (theory), anything that goes through a gradual transition from one condition, to a different condition, without any abrupt changes or "discontinuities" ofincreasing quality of performance and this was entirely consistent withthe original concept of criterion referencing. Glaser Noun 1. Glaser - United States physicist who invented the bubble chamber to study subatomic particles (born in 1926)Donald Arthur Glaser, Donald Glaser (1981), whoconceived criterion-referenced interpretation, agreed that the conceptincorporated a competence decision because it was about the ability toprogress along a continuum of increasing competence as defined byperformance quality. Standards referencing is recent terminology for aform of criterion-referenced interpretation. It is important to notealso that criterion referencing is not an assessment method. It is not atesting procedure; rather, it is the use of a defined interpretationframework. The correct use of the expression should refer tocriterion-referenced interpretation of assessment orstandards-referenced interpretation of assessment, notcriterion-referenced or standards-referenced assessment or testing inand of itself. In the early 1980s, the development of subject profiles within theschool systems was an extensive and practical approach to thedevelopment of criterion-referenced interpretation frameworks. This ledto the implementation of national profiles and curriculum statements andwithin a few years they became the basis of 'outcomes-basededucation'. Outcomes were defined in terms of increasing levels ofcompetence within discipline areas in the school curriculum. Outcomeswere described using 'progress maps', but progress maps couldbe generated by a small number of people capable of conducting itemresponse modelling analyses. Standards referencing was first proposed inQueensland Queensland,state (1991 pop. 2,477,152), 667,000 sq mi (1,727,200 sq km), NE Australia. Brisbane is the capital; other important cities are Gold Coast, Toowoomba, Townsville, Rockhampton, Cairns, and Ipswich. in the late 1980s and early 1990s (Sadler Sadler may refer to: Sadler, Texas, a US city Sadler and Company, a publisher of schoolbooks Sadler (surname), people with the surname Sadler See alsoSaddler , 1987) and gainedcredibility with the McGaw (1997) and later Masters (2002) reportsregarding the New South Wales New South Wales,state (1991 pop. 5,164,549), 309,443 sq mi (801,457 sq km), SE Australia. It is bounded on the E by the Pacific Ocean. Sydney is the capital. The other principal urban centers are Newcastle, Wagga Wagga, Lismore, Wollongong, and Broken Hill. Higher School Certificate This article is about the New South Wales Higher School Certificate. For the former British qualification, see Higher School Certificate (UK). The Higher School Certificate, or HSC . Largely throughthese reports, standards referencing became synonymous with synonymous withadjective equivalent to, the same as, identical to, similar to, identified with, equal to, tantamount to, interchangeable with, one and the same as outcomes-based education. Outcomes and competence-based education mergedthrough this terminology and standards referencing began to subsume sub��sume?tr.v. sub��sumed, sub��sum��ing, sub��sumesTo classify, include, or incorporate in a more comprehensive category or under a general principle: theideas of competency in school education. Education began to report in terms of developmental frameworksusing descriptions of skill levels and competencies; hence, if a similarshift occurred and a criterion or standards-referenced framework were tobe adopted for competency-based education in VET, recording methodswould need to be adjusted so that the records of achievement by studentsin 'VET in schools' programs could record their level ofperformance in addition to the competence dichotomy. Universities make decisions about which students to select intotheir courses and most use a ranking system based on Year Twelveexaminations. The system is usually based on a percentile rank The percentile rank of a score is the percentage of scores in its frequency distribution which are lower. For example, a test score which is greater than 85% of the scores of people taking the test is said to be at the 85th percentile. reportedat an assumed accuracy of two decimal places decimal placen.The position of a digit to the right of a decimal point, usually identified by successive ascending ordinal numbers with the digit immediately to the right of the decimal point being first: . Errors of measurement mustbe minimised and the quality and reliability of the assessment data needto be high in order to allow fine discrimination between candidatesseeking entrance into university. The current understanding of recognition requires that a person bedescribed as having achieved a status of competent or not-yet-competentfor units and elements in a training package. It is simply a two-levelcontinuum. If the continuum were to be expanded beyond the two levels toidentify how well a student had performed, observers would be able todifferentiate among those students classified as competent. Suchdifferentiation would not remove or invalidate in��val��i��date?tr.v. in��val��i��dat��ed, in��val��i��dat��ing, in��val��i��datesTo make invalid; nullify.in��val the current methods ofreporting and recording of competent and not-yet-competent, but it wouldprovide for extensions of this classification system to allow for thequality of the performance to be recognised and reported and then makeit possible to rank students in the same way that examinations at theend of Year Twelve enable them to be ranked for selection purposes.Moreover, opportunities heretofore denied to VET students would beopened up if VET subjects could be used for university entrance and inother selection contexts. Stringent requirements regarding themeasurement details of selection procedures would be necessary, however,if the recognition of competence were to be supplemented by the abilityto differentiate for selection purposes. The notion of competence The requirements necessary to supplement the assessment ofcompetence with differential grading can be achieved if we allow a smalladjustment in our understanding of competence. Competence has beengenerally defined as the capacity to meet the standard of performanceexpected in the workplace. This has been a fine definition but it is atruism that there is no fixed standard expected across workplaces withinan industry. Employers exploit their competitive advantage by arguingthat their workers are able to demonstrate superior performance againstthe competencies in training packages and that they expect and achievehigher standards of production than their competitors (Curwood, 2006).It is difficult to reconcile employers' recognition ofdifferentiated levels of performance with the argument that there is asingle defining point on any continuum to indicate when competence (thestandard expected in the workplace) has been achieved for a unit ofcompetence that defines skills required across workplaces with variableexpectations. Hence, regardless of whether there were two levels definedas competent and not-yet-competent, the continuum would not have asingle stable and invariant (programming) invariant - A rule, such as the ordering of an ordered list or heap, that applies throughout the life of a data structure or procedure. Each change to the data structure must maintain the correctness of the invariant. cut point across all workplaces in the sameindustry even for the same unit of competence. There is therefore a need for a different view of competence.Competence could be a person's capacity to adjust their performanceto meet the variable demands expected in workplaces. This incorporatesthe previous idea that a person can meet the standard expected in aworkplace, but it also says that the person experiencing differentworkplace expectations can adjust their level of performance to thestandards encountered across different workplaces or even within thesame workplace over time. This idea of competence incorporates the ideaof variable expectations and levels of performance quality and allowscompetence to be a construct used to interpret the quality ofperformance on a coherent series of tasks such as those found in thetraining packages. Assumptions This does not mean that it is necessary to jettison jettison(jĕt`əsən, –zən)[O.Fr.,=throwing], in maritime law, casting all or part of a ship's cargo overboard to lighten the vessel or to meet some danger, such as fire. the idea ofcompetence as a dichotomy or the practice of confirming competence onthe basis of performance on discrete tasks as long as the quality of theperformance can be taken into account. There is, however, a need for thetasks to be a coherent set ordered in terms of skill demand if they areto define a continuum of competence. It also requires a set ofassumptions to be explicated. They also apply to the dichotomy ofcompetence but are almost never considered (Griffin, 1997). The set of assumptions begins with the postulation that continua con��tin��u��a?n.A plural of continuum. can be constructed to describe development or growth in specific domainsof performance. The continua do not exist in and of themselves, but they areconstructed to assist in explaining observations of performance. Each continuum can be defined by a set of indicative tasksrepresenting levels of performance quality. Not all competencies can be directly observed. Related, indirectbehaviours can also be used, along with the directly observable ob��serv��a��ble?adj.1. Possible to observe: observable phenomena; an observable change in demeanor.See Synonyms at noticeable.2. behaviours. The indicators (behaviours) may be ordered along a continuumaccording to according toprep.1. As stated or indicated by; on the authority of: according to historians.2. In keeping with: according to instructions.3. the amount of competence required to demonstrate relativeperformance quality on each task. People can be ordered along the continuum according to theperformances they are able to exhibit. The continuum can be defined by any representative, cohesive cohesive,n the capability to cohere or stick together to form a mass. sampleof (performance quality) indicators that covers a range of levels on thecontinuum. Once the continuum is identified together with the underpinning un��der��pin��ning?n.1. Material or masonry used to support a structure, such as a wall.2. A support or foundation. Often used in the plural.3. Informal The human legs. Often used in the plural. construct, samples of indicative performances can be interchanged. The existence of higher order indicators implies the ability todemonstrate lower order behaviours. The relationship is probabilistic (probability) probabilistic - Relating to, or governed by, probability. The behaviour of a probabilistic system cannot be predicted exactly but the probability of certain behaviours is known. Such systems may be simulated using pseudorandom numbers. ,not causal causal/cau��sal/ (kaw��z'l) pertaining to, involving, or indicating a cause. causalrelating to or emanating from cause. . These assumptions define the idea of competence incorporatingvariable expectations and levels of performance quality. The developmental continuum There is an advantage to this way of defining the developmentalcontinuum. The theory underpinning item response modelling analysesshows that when the competence of the person equals the demands of theperformance (that is, both are located at the same point on thecontinuum) the odds of success are fifty-fifty. From this it can bededuced that, if the person were to improve a little, he or she wouldhave a better than even chance of demonstrating performances at or nearthat point on the continuum. It could be further argued that the mainoutcome of training is to increase the odds of success in each of theseworkplace performances. Levels on the continuum are defined by theclusters of groups of performance descriptions that have similar levelsof difficulty. As a person moves from one level to another on thecontinuum, the odds of fifty-fifty at the transition points can belinked to a change in the required performance quality and this can bedirectly translated into an implication for training. If the skillchanged, then this had an implication for a change in training contentand strategy. Clearly a fifty-fifty chance of success is unsatisfactoryfor the workplace. Surely no employer would want a worker who could onlyperform at a required level half the time. Consequently in any empiricalstudy of competence, it would be advisable ad��vis��a��ble?adj.Worthy of being recommended or suggested; prudent.ad��visa��bil to set item responsemodelling calibrations at a probability of a value such as 0.95 forsuccess. If the qualitative analysis Qualitative AnalysisSecurities analysis that uses subjective judgment based on nonquantifiable information, such as management expertise, industry cycles, strength of research and development, and labor relations. undertaken on the item responsemodelling results 'back translates' to match or closelyapproximate an original proposed idea of a developmental continuum itcan also be used as evidence of validity. The technique of'levels' has so far been used sparingly spar��ing?adj.1. Given to or marked by prudence and restraint in the use of material resources.2. Deficient or limited in quantity, fullness, or extent.3. Forbearing; lenient. but is increasinglyemerging in international studies; for example, Greaney, Khandker andAlam (language) ALAM - A language for symbolic mathematics, especially General Relativity.See also CLAM.["ALAM Programmer's Manual", Ray D'Inverno, 1970]. (1999) used the procedure in their report on the 'EducationFor All' project. Changed focus for 'VET in schools' program If the different status and esteem of 'VET in schools'subjects and mainstream academic subjects were to be removed, VETsubjects could be made available to all students and included in theprocedures used to calculate university entrance scores. Their exclusionhas tended to lower the esteem of VET subjects, despite considerableefforts having been made in many education jurisdictions to expand therange of VET subjects taken as equivalent to academic mainstreamsubjects. This could ensure that every Year Twelve subject was at leasteligible to contribute to university entrance scores so that studentswere not forced into life-changing decisions based on which subjectsthey might study and which career paths were open or closed to them. Asa consequence of the change in focus, a change in record keeping andreporting would be required to monitor students' levels ofcompetence as well as the competency dichotomy. The study The study addressed the issue of differentiation among students onthe basis of relative quality of performance and was conducted in fourindustries using a total of fifty-six competency units: seventeen unitsin Metal and Engineering, fifteen units in Information Technology,twelve units in Business Administration and fourteen competency units inHospitality. Sixty schools nationally participated in the project on theadvice of each of the state vocational education jurisdictions. Therewere nine schools in the Australian Capital Territory There are numerous public and non-governmental schools in the Australian Capital Territory. Children begin formal schooling at primary school in February when they are five years old (if they are due to turn five by 30 April in that year). , twelve in NewSouth Wales, five in Queensland, six in South Australia South Australia,state (1991 pop. 1,236,623), 380,070 sq mi (984,381 sq km), S central Australia. It is bounded on the S by the Indian Ocean. Kangaroo Island and many smaller islands off the south coast are included in the state. , thirteen inVictoria, five in Tasmania Tasmania(tăzmā`nēə), island state (1991 pop. 359,286), 26,383 sq mi (68,332 sq km), SE Commonwealth of Australia. It is separated from Australia by the Bass Strait and lies 150 mi (240 km) south of the state of Victoria. and ten in Western Australia Western Australia,state (1991 pop. 1,409,965), 975,920 sq mi (2,527,633 sq km), Australia, comprising the entire western part of the continent. It is bounded on the N, W, and S by the Indian Ocean. Perth is the capital. . The schoolsfocused on VET subjects and units of competency distributed over thefour industries. A total of 3,672 students were assessed, 493 in Metals,1,215 in Hospitality, 884 in Business and 1,080 in InformationTechnology. The students were assessed against each of a series of unitsof competence in their relevant VET study area. The assessments wereconducted by teachers who were provided with a rating sheet on whichthey recorded their observations of the relative quality of the studentperformance in the tasks associated with each of the units ofcompetency. This was then developed into a score for the subject. Thederivation derivation,in grammar: see inflection. of the score is described below. Developing the descriptions of competence A panel of experts was established for each of the VET study areas.The panels consisted of specialists nominated nom��i��nate?tr.v. nom��i��nat��ed, nom��i��nat��ing, nom��i��nates1. To propose by name as a candidate, especially for election.2. To designate or appoint to an office, responsibility, or honor. by the relevant nationalindustry training advisory boards. The panellists attended a workshoporganised by the project staff to explore the structure and developmentprocess for establishing rubrics that reflected relative quality ofperformance. Figure 1 depicts the structure of this process. [FIGURE 1 OMITTED] Training packages consist of units of competence. Each unitcontains elements of competence; each element was identified by a seriesof performance indicators. If a rubric RUBRIC, civil law. The title or inscription of any law or statute, because the copyists formerly drew and painted the title of laws and statutes rubro colore, in red letters. Ayl. Pand. B. 1, t. 8; Diet. do Juris. h.t. were developed for eachperformance indicator to indicate how well each task was completed, theissue of 'how well' could be addressed. The descriptions of'how well' were called quality criteria. Some performanceindicators might have two identifiable quality criteria, some three,some four, but in all cases a set of procedures for defining thecriteria would have to be followed. When a set of quality criteria werecombined with the performance indicators as rating scales the compositeis called a rubric. The panellists were shown how to write rubrics according to therules that quality criteria should: reflect levels of [workplace]performance quality not procedural steps; avoid counts of things rightor wrong; avoid all comparative terms; discriminate dis��crim��i��nate?v. dis��crim��i��nat��ed, dis��crim��i��nat��ing, dis��crim��i��natesv.intr.1. a. among orderedquality performances; enable assessees to verify (1) To prove the correctness of data.(2) In data entry operations, to compare the keystrokes of a second operator with the data entered by the first operator to ensure that the data were typed in accurately. See validate. their performanceassessment; focus on a series of single performances within the samedomain; and yield reliable and consistent judgments (Griffin, 1997). The number of quality criteria for each rubric had to be defined bythe panellists, who could each call on expertise outside the sphere oftheir own experience in order to avoid the restrictions of selfreferencing. Score development The model used for the study consisted of several steps. The firststep was defining the framework or the broad area of study, as shown onthe left in Figure 1. In this case it represented the training packageitself and represented the overall VET study area within which theassessment was conducted. The second level of the model described the key performance areas,or skill sets required within the framework. In the training packagethese were called the 'units of competence'. The third level defined the expectations employers had of assesseeswithin each skills set; that is, defined the assessees'capabilities through tasks which assessees needed to complete todemonstrate skills. In the training package these were called 'theelements'. The fourth level of the model was an evidence checklist, whichconsisted of a series of performance indicators. The checklist containeda series of typical or indicative tasks or performances that theassessors used to monitor the existence of competence, which theyrecorded as either achieved or not achieved ('competent' or'not yet competent').The assessee As`sess`ee´n. 1. One who is assessed.Noun 1. assessee - a person (or property) that is assessedindividual, mortal, person, somebody, someone, soul - a human being; "there was too much for one person to do" was expected to demonstrateall the indicative behaviours in order to be declared competent. The fifth and final step asked the question 'how well were theindicative tasks performed?' These descriptions were termed'quality criteria'. Existing competency-assessment documents already defined the firstfour levels: the framework (training package), key performance areas(units), expectations (elements), and evidence (performance indicators).Panellists in this study were required to define the fifth level, thequality criteria, in order to indicate how well each performanceindicator might have been performed by assessees of different capabilitylevels. The panellists did not have to weight the criteria, they onlyhad to define them using the rubric rules. As this was the first timesuch an exercise had been attempted, there was uncertainty among thepanellists regarding whether there was a developmental sequence amongthe indicators for any specific performance indicator. In some casesthere was a tendency to use 'steps taken' as indicators ofquality and this was discussed and remedied. An example is shown in Figure 2. The figure shows the model for ahospitality training package. One key performance area, or skillrequired, is for the assessee to 'work with colleagues andcustomers'; one expectation, or capability, as part of this skillis for the assessee to 'work in a team'; and one performanceindicator, or evidence, of this capability, is for the assessee to'recognise and accommodate cultural differences within theteam'. The panel identified three levels of quality of performance.These were written in the form of a multiple choice item, as shown inFigure 2, and the teachers were asked to select the option that bestdescribed the student's level of performance quality. [FIGURE 2 OMITTED] For the units involved in the Year Twelve assessment study, therewere fourteen expectations (or capabilities), and a total of twenty-sixperformance indicators. The panellists established a series of qualitycriteria for each performance indicator. There was no need to weight the criteria based on importance orcriticality as proposed by Thompson Thompson, city, CanadaThompson,city (1991 pop. 14,977), central Man., Canada, on the Burntwood River. A mining town, it developed after large nickel deposits were discovered in the area in 1956. , Mathers Mathers is a surname, and may refer to Marshall Mathers Debbie Mathers Edward Powys Mathers George Mathers James Mathers Jerry Mathers Jonathan Ian Mathers Kim Mathers Peter Mathers Richard Mathers Samuel Liddell MacGregor Mathers and Quirk quirk?n.1. A peculiarity of behavior; an idiosyncrasy: "Every man had his own quirks and twists"Harriet Beecher Stowe.2. (1996). Weightingof the quality criteria was based on the use of item response modellingwhich weights criteria (or score points) according to their capacity todifferentiate or discriminate between students. If the ratings were simply added across indicators within a unitand then across units within a subject, the performance indicators withthe largest number of quality criteria would have made the greatestcontribution to a total score. The subject with the most key performanceareas, expectations, performance indicators and quality criteria wouldscore highest, and the relative importance of a unit could be enhancedsimply by defining a larger number of quality criteria. This approach,in addition to being flawed flaw?1?n.1. An imperfection, often concealed, that impairs soundness: a flaw in the crystal that caused it to shatter.See Synonyms at blemish.2. , would generally lead to an equallyincorrect practice of insisting that all criteria had the same number oflevels or score points and it would require an irrelevant method ofweighting process to establish the relative importance of the rubrics.If there were more levels of quality, the importance or influence of thecriterion would be increased. Applying the logic of item responsemodelling was an important procedure in making this assessment approachaccessible. It also provided a way of defining the underlying continuumin terms of a developing competence linked to the training package. Adifferentiating score was a required outcome in this study and it madesense to use a weighting method directly related to that purpose. In this project the scores within performance indicators wereweighted on the basis of their effectiveness to differentiate betweenstudents. This meant that only the very best students would achieve thehigher scores on the most discriminating dis��crim��i��nat��ing?adj.1. a. Able to recognize or draw fine distinctions; perceptive.b. Showing careful judgment or fine taste: items. So a simple addition ofperformance indicator scores could be used because it would produce adiscriminating total score. It would then be possible to standardise Verb 1. standardise - evaluate by comparing with a standardstandardizeappraise, assess, evaluate, valuate, value, measure - evaluate or estimate the nature, quality, ability, extent, or significance of; "I will have the family jewels appraised by a thesubject scores to produce a university entrance score (where scores wererequired) or, in the case of Queensland, an overall position (OP) band,(Griffin, Gillis, Keating, & Fennessy, 2001). The results suit the selection process by providing ranked scores;however, their usefulness is limited in a VET competency context. Thenumbers have little substantive meaning unless they are linked to thecompetency criteria they are meant to represent. The debate on grading,while antithetical an��ti��thet��i��cal? also an��ti��thet��icadj.1. Of, relating to, or marked by antithesis.2. Being in diametrical opposition. See Synonyms at opposite. to the idea of standards referencing, posited thepossibility of using a system of score reporting that looked likegrading but which described in performance terms how well a person hasperformed in a unit or a subject. Grading is not an assessment method. It is a normative nor��ma��tive?adj.Of, relating to, or prescribing a norm or standard: normative grammar.nor method ofreporting that translates into a letter grade ranging from A to F, forexample, determined by a distribution with no real substantiveinterpretation of the letter grades other than relative locations in thedistribution. Like scores, grading too has limited usefulness in a VETcompetency context. By comparison, a standards-referenced framework hasno a priori a prioriIn epistemology, knowledge that is independent of all particular experiences, as opposed to a posteriori (or empirical) knowledge, which derives from experience. distribution across the levels. Ideally, all students coulddemonstrate performance at the highest possible level. It has a greaterpotential utility in a VET competency context. The differentiating score, developed by the procedure describedabove, was based on recognising the amount of skill required to performa task at a specified level of performance quality. For each performancetask, a behavioural Adj. 1. behavioural - of or relating to behavior; "behavioral sciences"behavioral rating scale was used, usually with a score out oftwo, three or four with each additional score point recognising anincreasing level of performance quality. The higher the score, the moredifficult it was to perform the task at that level of quality; however,not all equal score numbers represented the same relative level ofquality. A score of three for one criterion might be very difficult toobtain whereas the same score of three on another criterion might berelatively simple to obtain. The score of three that was more difficultto obtain therefore identified more capable people and generally onlythe capable people could attain this score level of quality. The scoreon the easier criterion did not separate out the more capable people butdid identify and describe the highest level of performance for that taskand helped to identify and define the performances of students at lowerlevels of quality and at a lower capability of competence. The task forthe panel members was to define the performance levels associated witheach score point for each performance task and to compare each criterionwith every other criterion within a unit of competence in order toestablish the discriminating power of the criterion. The panel members used a grid to differentiate between performancequality levels. Each column in the grid In the Grid is a game show that airs on UK broadcaster Five at 6.30pm week nights. It first aired on Monday 30 October 2006.In the Grid is hosted by Les Dennis and is produced by Initial West, one of the Endemol UK companies. represented a task. Rowsrepresented the relative difficulty of performances. The higher the rowthe more difficult it was to demonstrate that level of performance.Numbers were placed in the vertical columns at a row height thatrepresented the relative difficulty of performing the task at that levelof quality. The row height in the column indicated how difficult it wasto achieve that score. The most difficult performances were at the topand the easiest ones were at the bottom of the grid. When all codes wereplaced in the vertical space, the scores were replaced by thedescriptions of the performances that each score represented. The panellists identified horizontal groupings or clusters ofcriteria that were at or about the same height in the grid. Theyinterpreted those clusters and wrote a description of the general themethey represented. The themes were refined into descriptions of thelevels of competence that students demonstrated in each competency unit. The theme descriptions were at times quite long so summary versionsof each overall level description were developed. The summary statementsin turn became levels in a rating scale that could be used to recordperformances across multiple units. This was an important step becausethe summary statements were then conflated across units to obtain scoresfor subjects. This in turn enabled a shift in the focus of theassessment from performance indicators to units of competence and thento obtain a score for a VET subject, and yielded a score for a subjector certificate from a combination of units. These summary statements andthe resultant This article is about the resultant of polynomials. For the result of adding two or more vectors, see Parallelogram rule. For the technique in organ building, see Resultant (organ).In mathematics, the resultant of two monic polynomials record sheet are shown in Figure 3. [FIGURE 3 OMITTED] It was possible to provide a report for students for eachcompetency unit, indicating both the level of performance quality and ascore. The teacher needed to rate each student on each performanceindicator to obtain a unit score and competency level and then recordthe competency level. Instead of ticking tickinga coat color pigmentation pattern in which hairs of one color are distributed in small groups throughout the background color, e.g. Australian cattle dog. Called also speckling. as each was observed, theteacher selected a description of the quality criterion that bestmatched the student performances. This ensured that there was noadditional workload The term workload can refer to a number of different yet related entities. An amount of laborWhile a precise definition of a workload is elusive, a commonly accepted definition is the hypothetical relationship between a group or individual human operator and task demands. for teachers in recording the assessment. The result was a school-based assessment for 'VET inschools' subjects that could be subjected to statistical andconsensus moderation procedures, be differentially weighted,standardised Adj. 1. standardised - brought into conformity with a standard; "standardized education"standardizedstandard - conforming to or constituting a standard of measurement or value; or of the usual or regularized or accepted kind; "windows of standard width"; and scaled to produce the universities admission indexappropriate for each of the national education jurisdictions. Eachjurisdiction would be able to use a moderated score system that wouldenable a scaled university entrance score to be obtained. The scores and the score range for units in thestandards-referenced framework are shown in Table 1. The example isbased on an analysis of the fourteen units common to the New South WalesHigher School Certificate and the Victorian Certificate of Education The Victorian Certificate of Education (VCE) is the credential given to students who have completed Year 11 and Year 12 of their secondary schooling, in the state of Victoria, Australia. inthe Hospitality subject. The distribution does not represent any overallstate distribution of student scores because a calibration calibration/cal��i��bra��tion/ (kal?i-bra��shun) determination of the accuracy of an instrument, usually by measurement of its variation from a standard, to ascertain necessary correction factors. sample wasused to establish the properties of the standards-referenced frameworkand the calibration sample was not a representative random sample ofstudents. Data collected from the project were merged with each statecentral examination data and it helped to illustrate that the procedurecould be carried out and that statistical moderation of the school-basedassessment is possible with item response modelling analysis. From Table 2 it can be seen that there were variable numbers ofperformance indicators for each unit. Ni, the number of performanceindicators within a competency unit, ranged from twenty-six indicators(unit THHCOR01B) to six (unit THHGHS01B). Measures of reliability wereobtained using the scores assigned as��sign?tr.v. as��signed, as��sign��ing, as��signs1. To set apart for a particular purpose; designate: assigned a day for the inspection.2. to indicators within units and ascore from which levels of competence could be identified. Reliabilityestimates ranged from 0.95 to 0.83. Maximum scores for each of the unitsare also shown and these were obtained by summing the maximum qualitycriteria ratings over all indicators within a unit. Panellists definedthe cut scores for the change of competence level. The summary descriptions for each level within a unit shown inFigure 3 enable an aggregation process to be used to obtain a subjectscore. A student judged across the fourteen units being at competencelevels (4,3,3,2,3,3,2,2,3,2,3, 2,2,3) would gain an aggregate subjectscore of thirty-seven from a maximum possible of forty-three. From Table3 it can be seen that the overall reliability of the scores for theaggregation of units was alpha = 0.84. The mean score and standarddeviation In statistics, the average amount a number varies from the average number in a series of numbers. (statistics) standard deviation - (SD) A measure of the range of values in a set of numbers. were 9.35 and 8.10 respectively in a possible range from zeroto forty-three. These measures do not have a contribution to make to theinterpretation of competence, except to illustrate that there was a widespread of scores and most were about the mid range of possibleperformance. It was possible, however, to grade the scores on a scale Ato F using cut scores as shown in Table 3. Given the failure of scoresto communicate the competency levels of the candidates, it was necessaryto translate the grades into competency statements and have one levelfor each unit identified by the expert panel as the minimum forrecognition of competence in that unit. In Figure 3 the first level isbelow the minimum level. Consistency Classical measures of consistency include the indices ofreliability known as Cronbach's alpha Cronbach's (alpha) has an important use as a measure of the reliability of a psychometric instrument. It was first named as alpha by Cronbach (1951), as he had intended to continue with further instruments. . This index ranges in valuefrom 0.0 to 1.0. Zero indicates that there is no consistency and thatthe assessment is entirely random. More recently, item response theories Item response theory is a body of theory used in the field of psychometrics. Pychometrics is concerned with the theory and technique of educational and psychological measurement. have added indices of 'separation' reliability and theyprovide some interesting interpretations. An item separation reliabilityindex is closely related to the Cronbach index but indicates the extentto which the items or, in this case, the rubrics or quality criteriawere separated along the underpinning construct or continuum. A value ofzero indicates that the rubrics are all clustered at the same point andeach indicates the same level of performance. Adding more items orrubrics at the same level does not add value to the assessment. As theitems are increasingly separated, the value of the index rises to 1.0.At 1.0 the rubrics or quality criteria are completely separated and eachone adds new meaning to the nature of the continuum and the competencyconstruct. Given that they add to the meaning of the construct, thesedata can also be interpreted as a measure of the validity of theconstruct. Item response theory also provides a measure of personseparation. The index ranges from 0.0 to 1.0 with similar meaning to thevalues for the item separation index. In the case of persons, however,perfect separation on the underlying variable would indicate that it ispossible to accurately identify differences between the personsassessed. It is possible to use this as a measure of criterion validity The introduction to this article provides insufficient context for those unfamiliar with the subject matter.Please help [ improve the introduction] to meet Wikipedia's layout standards. You can discuss the issue on the talk page. .Item response modelling therefore tends to merge the ideas ofreliability of data and validity of interpretation in ways that otherapproaches to assessment cannot. There was also another form of consistency needed in this project.The expert panel judgments of the indicator difficulty level needed tobe consistent with the empirical calibration of the level of theindicator. That is, the validity and consistency of the panel judgmentneeded to be verified ver��i��fy?tr.v. ver��i��fied, ver��i��fy��ing, ver��i��fies1. To prove the truth of by presentation of evidence or testimony; substantiate.2. . The validity and reliability relied on the panelapproach yielding the same definition of the continuum as yielded by theitem response modelling analysis. This was tested by comparing the panelplacement of rubrics at levels of performance and then empiricallycalibrating their relative difficulty using item response modelling anddata collected by teacher judgments of student performances. A standarderror of judgement was defined as the standard deviation of thedifferences between the panellists and item response modellingapproaches taken over judgments within a unit of competence. There is aneed for further research to see if this measure was appropriate and todetermine how to assess the significance of its value. The standarderror of judgement for the Hospitality data used in this project, forexample, was 0.27. Zero would indicate a perfect match, but it is notknown what the overall distribution statistics are for the standarderror of judgement and hence it is not possible to state how impressiveor otherwise this figure is. Given the small value, it does suggest thatthere was a close match and that the subject matter experts group couldbe regarded as making valid judgments of relative levels of performancequality and task difficulty. In Table 3, the reliability indices arereported for Hospitality assessments and in Victoria and NSW NSWNew South WalesNoun 1. NSW - the agency that provides units to conduct unconventional and counter-guerilla warfareNaval Special Warfare , where theassessments are combined with central examinations, the combinedreliabilities were also assessed, the reliabilities for thestandards-referenced framework were not state based and only nationalreliability is reported for these measures. The standards-referenced framework developed in this study had highreliability and validity indices, lending considerable support for itsuse, whether alone or in combination with a central exam or other formsof assessment. It is likely that the use of a mean square error willultimately yield a measure of agreement and consistency of thepanellists' estimates. Procedures reported by Bateman Bateman might refer to: PeopleCharles Bateman, architect C. Donald Bateman Chief Engineer, Flight Safety Avionics, Honeywell, Bellevue, WA. Inventor of EGPWS and older obsolete GPWS. (2003) andConnally (2004) relied on additional judgment to assess the level ofagreement and were based more on intuition intuition,in philosophy, way of knowing directly; immediate apprehension. The Greeks understood intuition to be the grasp of universal principles by the intelligence (nous), as distinguished from the fleeting impressions of the senses. than measurement. Consistency is an issue that has to be addressed. In makingdecisions about competence, there is a need to not only be consistentbut to appear to be as well. The following analyses show that, for themost part, the competency assessment for a subject or for adifferentiating score is unaffected by the selection of units or by howmany units are used. Figure 4 relates student ability or competence tothe standards-referenced band assigned for the each unit. The closealignment of the lines shows that there is little difference in which orhow many units are used to assess overall competence. [FIGURE 4 OMITTED] Figure 5, however, shows that the assessment can differconsiderably depending on the location of the assessment. In theAustralian Capital Territory Australian Capital Territory(1991 pop. 276,468), 939 sq mi (2,432 sq km), SE Australia, an enclave within New South Wales, containing Canberra, capital of Australia. It was called the Federal Capital Territory until 1938. , for example, a student had to demonstratea high level of performance quality in order to be assessed as competent(represented by the horizontal line (Descriptive Geometry & Drawing) a constructive line, either drawn or imagined, which passes through the point of sight, and is the chief line in the projection upon which all verticals are fixed, and upon which all vanishing points are found.See also: Horizontal at a value of 1.0.) In most otherstates, a much lower level of performance quality was required in orderfor the student to be assessed as competent. In this study, thedifference due to location was controlled through the application ofitem response modelling analysis but in most assessments, where theweighted differentiating score is not used and a decision of competenceis made, this difference is uncontrolled and the consistency ofdecision-making decision-making,n the process of coming to a conclusion or making a judgment.decision-making, evidence-based,n a type of informal decision-making that combines clinical expertise, patient concerns, and evidence gathered from is variable. This underpins the point that there is nofixed level of competence. Not only is the level of performance expectedin the workplace variable, but the competence decision varies accordingto the demands of the curriculum in the school system. The importantthing is whether the student can meet the expectations of the workplaceor the school system. This analysis has exposed the variability inexpectations demanded for attributing competence. [FIGURE 5 OMITTED] Figures 4 and 5 show that there were different views of competenceacross systems and action needs to be taken to address this lack ofconsistency. The lack of consistency was not an artefact See artifact. of the currentprocedure demonstrated in this article; it was a hidden aspect ofcompetency assessment, which relied on judgment in context. We expectthe effect would be exacerbated across workplaces and across assessors.The lack of consistency is not a weakness of the proposed assessment ofcompetence and quality; on the contrary, the identification of thisinconsistency in��con��sis��ten��cy?n. pl. in��con��sis��ten��cies1. The state or quality of being inconsistent.2. Something inconsistent: many inconsistencies in your proposal. is a strength of this analysis. Previous investigations of consistency of competency assessmenthave not focused on outcomes or on the performances of the assessees(Victorian Victorianone reflecting an unshaken confidence in piety and temperance, as during Queen Victoria’s reign. [Am. and Br. Usage: Misc.]See : Prudery Education Training and Assessment Services, 2000). They tendto have examined the procedures and materials. Even with constantprocess and materials, differences exist in the interpretation ofcompetence. If a consistent national process were to be used it would bepossible for national standards to be set and monitored in competencyassessment and to control effects of location, localised localised - localisation curriculumeffects and judgment inconsistency. Hence, the method reported in thisarticle not only provides an opportunity for scored assessment, it addsthe notion of quality to competence and allows monitoring of standardsand the identification of any bias in competency assessment. Discussion In this study the capacity of the student or the level ofperformance has been controlled for the effect of the location. Withoutthe weighted differentiating score and item response modellingcalibration, this effect would have to be controlled through moderation.Consistency of competence assessment is an issue that still needs to beresolved. The methods displayed in this article have shown a possibleapproach to resolving or at least identifying such an issue. Finally the study has illustrated a series of principles that havebeen adhered to and that were recommended for the Australian system(Griffin, Gillis, & Calvitto, 2004). The procedures reported in this article offer an opportunity tocombine quality with competence and to encourage industry and trainingorganisations to pursue excellence through training and target settingwithin the training package frameworks. They have implications for arange of organisations and individuals. They require an acceptance oflevels of performance that go beyond the dichotomy of competent andnot-yet-competent but this requires an ideological shift that may beresisted. The approach is clearly differentiated from a normative approach tograding. It is also necessary to accept that a score could be obtainedfor each unit and for combinations of units making up a subject orcertificate. Scores for discrete tasks or performances were weighted onthe basis of their capacity to differentiate between students, ratherthan on importance, or the number of indicators or criteria. The scoringand weighting procedure automatically transform numerical numericalexpressed in numbers, i.e. Arabic numerals of 0 to 9 inclusive.numerical nomenclaturea numerical code is used to indicate the words, or other alphabetical signals, intended. scores into astandards-referenced framework whereas other approaches provide anormative distribution. The indicators, criteria and competency levelswere also based directly on training package competencies rather thangeneric skills or assessment methods and hence the validity of theassessment was enhanced (McCurry, 2003; Messick, 1992). As a result of this study a series of principles were proposed byGriffin and Gillis (2001). The principles underpinned the development ofthe standards-referenced framework and its attendant ATTENDANT. One who owes a duty or service to another, or in some sort depends upon him. Termes de la Ley, h.t. As to attendant terms, see Powell on Morts. Index, tit. Attendant term; Park on Dower, c. 1 7. reporting system. The principles stated that in developing quality-based competencyassessment, the following must be true. 1 The system of assessment and reporting must be situated in atheory of learning and assessment. 2 The procedure and assessment must satisfy both criterion- andnorm-referenced interpretation. 3 The model, approach used, assessment method, materials anddecisions must be transparent and externally verifiable through a formalaudit process. 4 The assessment procedure and the model must be resource-sensitivein both development and application. 5 The model and the approach to assessment and reporting mustaccommodate the existing assessment procedures that workplace assessorshave been trained to use with minimal change. 6 The model and its procedures should be accessible to subjectmatter experts. 7 The procedure must have both face and construct validity construct validity,n the degree to which an experimentally-determined definition matches the theoretical definition. . 8 The procedures must be demonstrably de��mon��stra��ble?adj.1. Capable of being demonstrated or proved: demonstrable truths.2. Obvious or apparent: demonstrable lies. fair, equitable equitableadj. 1) just, based on fairness and not legal technicalities. 2) refers to positive remedies (orders to do something, not money damages) employed by the courts to solve disputes or give relief. (See: equity) EQUITABLE. and unbiased. 9 The model must be communicative com��mu��ni��ca��tive?adj.1. Inclined to communicate readily; talkative.2. Of or relating to communication.com��mu and satisfy the information needsof stakeholders StakeholdersAll parties that have an interest, financial or otherwise, in a firm-stockholders, creditors, bondholders, employees, customers, management, the community, and the government. in a quality assurance context that must beaccommodated. 10 The scores and assessments must be amenable AMENABLE. Responsible; subject to answer in a court of justice liable to punishment. to statistical andor consensus moderation to ensure consistency of decisions and accuracyof score. This set of principles is recommended for a joint assessment ofboth quality and competence. Key wordsvocational educationcompetencyassessmentsecondary educationRasch modeluniversity selection References Bateman, A. (2003). A validation See validate. validation - The stage in the software life-cycle at the end of the development process where software is evaluated to ensure that it complies with the requirements. of multi source assessment ofhigher order competency assessment. Unpublished masters thesis, Facultyof Education, The University of Melbourne AsiaWeek is now discontinued. Comments:In 2006, Times Higher Education Supplement ranked the University of Melbourne 22nd in the world. Because of the drop in ranking, University of Melbourne is currently behind four Asian universities - Beijing University, , Australia. Connally, J. (2004). A multi source assessment of higher ordercompetencies. Unpublished doctoral thesis, Faculty of Education, TheUniversity of Melbourne, Australia. Curwood, M. (2006). A case study of the implementation ofcompetency based assessment and training in Australian industry.Unpublished doctoral thesis, Faculty of Education, The University ofMelbourne, Australia. Glaser, R. (1981). The future of testing: A research agenda forcognitive psychology cognitive psychology,school of psychology that examines internal mental processes such as problem solving, memory, and language. It had its foundations in the Gestalt psychology of Max Wertheimer, Wolfgang Köhler, and Kurt Koffka, and in the work of Jean and psychopathology psychopathology/psy��cho��pa��thol��o��gy/ (-pah-thol��ah-je)1. the branch of medicine dealing with the causes and processes of mental disorders.2. abnormal, maladaptive behavior or mental activity. , American Psychologist The American Psychologist is the official journal of the American Psychological Association. It contains archival documents and articles covering current issues in psychology, the science and practice of psychology, and psychology's contribution to public policy. , 36(9),9-23. Greaney, V., Khandker, S.R., & Alam, K. (1999). Bangladesh Bangladesh(bäng-lädĕsh`, băng–)[Bengali,=Bengal nation], officially People's Republic of Bangladesh, republic (2005 est. pop. 144,320,000), 55,126 sq mi (142,776 sq km), S Asia. :Assessing basic skills. Dhaka Dhakaor Dacca(both: dăk`ə), city (1991 pop. 6,844,131), capital of Bangladesh, on a channel of the Dhaleswari River, in the heart of the world's largest jute-growing region. : University Press. Griffin, P. (1995). Competency assessment: Avoiding the pitfalls ofthe past. Australian and New Zealand New Zealand(zē`lənd), island country (2005 est. pop. 4,035,000), 104,454 sq mi (270,534 sq km), in the S Pacific Ocean, over 1,000 mi (1,600 km) SE of Australia. The capital is Wellington; the largest city and leading port is Auckland. Journal of Vocational Education,3(2), 33-59. Griffin, P. (1997, September September:see month. 18). Developing assessment in schoolsand workplace. Paper presented at the Inaugural Professorial Lecture,Dean's Lecture Series, Faculty of Education, The University ofMelbourne. Griffin, P., & Gillis, S. (2001, May 4). Competence andquality: Can we assess both? Paper presented at the UpgradingAssessment: A National Conference on Graded Assessment, Kangan BatmanInstitute of TAFE TAFE(in Australia) Technical and Further Education , Melbourne Melbourne, city, AustraliaMelbourne,city (1991 pop. 2,761,995), capital of Victoria, SE Australia, on Port Phillip Bay at the mouth of the Yarra River. Melbourne, Australia's second largest city, is a rail and air hub and financial and commercial center. . Griffin, P., & Gillis, S. (2002). Scored assessment for Year12. (Report of the pilot study). Melbourne: The University of Melbourne,Assessment Research Centre. Griffin, P., Gillis, S., & Calvitto, L. (2004). Connectingcompetence and quality: Scored assessment in Year 12 VET. Melbourne: TheUniversity of Melbourne, Assessment Research Centre. Griffin, P., Gillis, S., Keating J., & Fennessy, D. (2001).Assessment and reporting of VET courses within senior secondarycertificates. In Creating expanded opportunity for youth: Greaterrecognition for VET courses in industry and university. Sydney Sydney, city, AustraliaSydney,city (1991 pop. 3,097,956), capital of New South Wales, SE Australia, surrounding Port Jackson inlet on the Pacific Ocean. Sydney is Australia's largest city, chief port, and main cultural and industrial center. : NewSouth Wales Department Vocational of Education and Training Masters, G.N. (2002). Fair and meaningful measures? A review ofexamination procedures in the NSW Higher School Certificate. Melbourne:Australian Council for Educational Research The Australian Council for Educational Research (ACER) is a non-governmental educational research organisation based in Camberwell, Victoria and with offices in Sydney, Brisbane, Perth, Dubai and India. . McCurry, D., (2003). But will it work in theory? Theory,empiricism empiricism(ĕmpĭr`ĭsĭzəm)[Gr.,=experience], philosophical doctrine that all knowledge is derived from experience. For most empiricists, experience includes inner experience—reflection upon the mind and its , pragmatics pragmaticsIn linguistics and philosophy, the study of the use of natural language in communication; more generally, the study of the relations between languages and their users. and the key competencies: the place of theory andresearch in the development of a notion of a work related skills and thewhole school assessment of generic skills. Melbourne: Australian Councilfor Educational Research. McGaw, B. (1997). Shaping their future: Recommendations for reformof the Higher School Certificate. Sydney: NSW Department of Training andEducation Coordination. Messick, S. (1992). The interplay in��ter��play?n.Reciprocal action and reaction; interaction.intr.v. in��ter��played, in��ter��play��ing, in��ter��playsTo act or react on each other; interact. of evidence and consequences inthe validation of performance assessments: Research report. Paperpresented to the Annual Meeting of the National Council on Measurementin Education, San Francisco San Francisco(săn frănsĭs`kō), city (1990 pop. 723,959), coextensive with San Francisco co., W Calif., on the tip of a peninsula between the Pacific Ocean and San Francisco Bay, which are connected by the strait known as the Golden , CL, USA. National Assessors and Workplace Trainers Body. (1998). Trainingpackage for assessment and workplace training. Melbourne: AustralianNational Training Authority. Sadler, R. (1987). Specifying and promulgating achievementstandards. Oxford Review of Education, 13(2), 191-209. Stanley, G. (1993). The psychology of competency based education.In C. Collins (Ed.), The debate on competencies in Australian educationand training. Canberra Canberra(kăn`bərə), city (1991 pop. 276,162), capital of Australia, in the Australian Capital Territory, SE Australia. The Canberra urban agglomeration includes a small area in New South Wales. : The Australian College of Education. Thomson, P., Mathers, R., & Quirk, R. (1996). Grade debate:should we grade competency-based assessment? Adelaide Adelaide, empress consort of Holy Roman Emperor Otto IAdelaide(ăd`əlād)or Adelheid(ä`dĕlhīt), c. : National Centrefor Vocational Education Research. Victorian Education Training and Assessment Services. (2000). VCE VCE Victorian Certificate of Education (State of Victoria, Australia)VCE Virginia Cooperative ExtensionVCE Volvo Construction EquipmentVCE Venice, Italy - Marco Polo (Airport Code)VET Graded Assessment Pilot Project 1999. Melbourne: Victorian Board ofStudies. Professor Patrick Griffin is Director of the Assessment ResearchCentre, Chair of Education (Assessment), Deputy Dean, and Associate Deanof Innovation and Development at the University of Melbourne. Email p.griffin@unimelb.edu See .edu. (networking) edu - ("education") The top-level domain for educational establishments in the USA (and some other countries). E.g. "mit.edu". The UK equivalent is "ac.uk". .au Dr Shelley Gillis is a lecturer lecturerA person who is primarily–if not entirely—involved in the teaching activities of an academic center, who is not expected to perform research or Pt management; in general, lectureships are non-tenured positions and research fellow at theAssessment Research Centre. Leanne Calvitto was a research associate at the Assessment ResearchCentre at the time of this study.Table 1 Distribution and score properties for subject aggregationHospitality Levels ->Unit Ni [alpha] m s MaxSchoolBased 14 0.84 9.35 8.10 44Hospitality F E D C B AUnit Level Max Cut ScoreSchoolBased 4 9 22 33 40 44Table 2 Distribution and score properties for Hospitality unitsHospitalityUnit Code Unit Description Ni ([alpha]) MaxTHHCOR01B Work with colleagues and customers. 26 0.95 50THHCOR02B Work in a socially diverse environment 10 0.87 19THHCOR03B Follow health, safety and security procedures. 11 0.86 18THHGHS01B Follow workplace hygiene procedures 6 0.83 15THHH0001B Develop and update hospitality industry knowledge 10 0.92 22THHGGA01B Communicate on the telephone. 13 0.90 24THHBH01B Provide housekeeping services to guests. 11 26THHBH03B Prepare room for guests 20 43THHBKA01B Organise and prepare food 16 0.92 36THHBKA02B Present food 10 0.90 24THHGFA01B Process financial transactions 15 0.95 35THHGCS02B Promote products and services to customers 12 30THHBFB10B Prepare and serve non alcoholic beverages 10 0.94 22THHBKA03B Receive and store kitchen supplies 15 0.93 26Hospitality 0 1 2 3 4Unit Code Level Max Raw ScoreTHHCOR01B 2 9 25 41 50THHCOR02B 3 7 12 19THHCOR03B 3 8 15 18THHGHS01B 3 9 12 15THHH0001B 2 7 15 22THHGGA01B 4 10 18 24THHBH01B 10 15 20 26THHBH03B 11 21 29 43THHBKA01B 3 6 20 32 36THHBKA02B 4 15 24THHGFA01B 6 11 20 35THHGCS02B 6 21 28 30THHBFB10B 7 12 17 22THHBKA03B 5 10 15 26([alpha]) Represents the reliability of the score for the unitTable 3 Reliabilities and distributions of the Hospitality assessmentsin NSW and Victoria Separation ScoreUnit * Alpha Item Case Mean SDSRF (all) 0.84 0.55 0.78 9.35 8.1SRF (Vic) -- -- -- 8.55 8.29SRF + FB Vic Exam 0.96 0.64 0.86 93.28 39.8SRF + CC Vic Exam 0.96 0.71 0.89 65.27 41.7FB Exam Only 0.95 0.86 0.94 98.51 34.7CC Exam Only 0.95 0.7 0.94 78.9 34SRF + NSW Exam 0.76 0.86 0.79 32.1 7.09SRF (NSW) -- -- -- 10.56 8.3NSW Exam Only 0.76 0.98 0.82 32.15 6.77 LevelUnit * F E D C B ASRF (all) 4 9 22 34 36 44SRF (Vic) 11 21 27 33 40SRF + FB Vic Exam 35 70 157 209 215SRF + CC Vic Exam 34 62 93 155 202 218FB Exam Only 22 47 126 175CC Exam Only 23 42 66 121 160 178SRF + NSW Exam 8 32 58 74 85 98SRF (NSW) 8 16 22 24 29NSW Exam Only 0 16 36 50 56 69* SRF = Standards Referenced Model; FB = Food and Beverage,CC = Commercial Cookery