Fresh starts are a feature of a test that can make the results more reliable. A test with fresh starts will be divided into a number of sections, each testing a different area of language or skills/subskills, so that no one area dominates. Consider a placement test that includes a speaking test where the learner is asked to talk about what she did for her last holiday and nothing else. Her final mark is liable to be dominated by her accuracy and fluency in using past verb forms, plus her knowledge of the specific lexis involved (eg connected with the sea, mountains, sightseeing or whatever the holiday involved). If the mark is high, it may hide the fact that when talking about the future she relies exclusively on will + infinitive, or if it is low, may not reveal that in general social conversation on other topics, she is both accurate and fluent and has a wider range of vocabulary.
On the other hand, a speaking test which started by asking her to talk about herself (where she's from, family, work, interests etc), then asked her to talk about what she did for her last holiday, and then asked her to talk about her plans for the next few months, would have a number of "fresh starts" - ie changes from one topic and area of language to another. If she showed weakness in the "holiday" section this time, but performed accurately and fluently in the others, her final mark would be much more representative of her overall competence.
So fresh starts lead to a much more reliable result - there is no chance of the result being swayed by a specific strength or weakness.
Showing posts with label Testing. Show all posts
Showing posts with label Testing. Show all posts
An ELT Glossary : Discrete Item and Integrative Tests, Direct and Indirect Testing
14. When I was a child I .......... in a choir.
a. sing b. singed c. song d. sung e. sang
They have the advantages of often being practical to administer and mark, and objective in terms of marking. However, they show only the learners ability to recognise or produce individual items - not how s/he would use the language in actual communication. In other words, they are inevitably indirect tests - they provide evidence of the learners' ability to recognise or produce certain specific elements of the language, but do not demonstrate how they might actually use them (or anything else) in communication. Learners’ abilities are inferred rather than demonstrated.
Integrative tests, on the other hand, may be either direct or indirect. The use of the term integrative indicates that they test more than one skill and/or item of knowledge at a time. Dictation is an integrative test, because it involves listening skills, writing skills, recognition of specific language items, grammar (eg in order to distinguish whether /əv/ should be written as have or of) and so on. Dictation is still, however, an indirect test,
Many integrative tests, on the other hand, are often also direct tests - they ask the learner to demonstrate their ability to perform a specific "real life" communicative task by asking them to actually do it. They therefore demonstrate the learners's ability to use the language in actual communication.
) “Whereas
discrete items attempt to test knowledge of language one bit at a time, integrativetests attempt to assess a learner's capacity to use many bits all at the same time, and possibly while exercisingseveral presumed components of a grammatical system, and perhaps more than one of the traditional skills oraspects of skills.
”
An example of a direct test which is also integrative might be role-playing a job interview, where the learners have to listen and understand the examiner's questions, choose the relevant grammar and lexis to express their ideas, speak with intelligible/accurate pronunciation and intonation etc. Other direct tests might include eg writing a letter of complaint;
or reading and completing an application form.These direct tests all demand that the learner integrates a variety of skills and items of linguistic knowledge to complete the test, and show how effectively the learner can use the language in communication. They have the disadvantage however, that they do not necessarily evidence knowledge of specific items - for instance, a learner might fail to produce a difficult structure by expressing the same concept in a different manner - eg I didn't know so I didn't go rather than If I had known I would have gone. With a direct test it is impossible to know whether the learners "avoided" the difficult structure because they had never met it or knew they would produce it inaccurately, or whether it was just by chance that they chose to say one thing rather than another. An indirect, discrete item test can, on the other hand, "push" the learner into demonstrating their knowledge (or lack of it) of chosen structure, lexis etc.
For this reason, many test batteries use a mixture of indirect tests and direct tests - each can balance the deficiencies of the other.
Recommended Further Reading
Hughes, A. Testing for Language Teachers, OUP
An ELT Glossary : Fit for Purpose
Some examples :
- if a progress test or an achievement test did not test the items taught in the preceding course or section of a course, the test would lack content validity and not be "fit for purpose". It would fail to show if the learners had or hadn't assimilated and retained those items and therefore would not accurately reflect their progress or achievement.
- if a placement test did not have a number of items likely to be known by learners who were at each level of linguistic competence (say for the CEFR levels A1 - C2) its coverage would not discriminate effectively between learners. If for example it had a very limited number of items at A2 and all of them at the upper end of the level, it could result in learners who were actually at B1 level being placed in A2 classes (because they had correctly answered the A1 items but not the "hard" A2 items or items from B1 and beyond). However, that learner might well have a good grasp of "easier" A2 items that were not included in the test and actually be ready to start a B1 class - where revision of "harder" A2 items would probably be included. It would therefore not be "fit" for its intended purpose of placing learners in the correct class.
- similarly, if learners wanting to enrol at a language school were told that the placement test (taken before they have made the final decision to enrol) would take three hours of their time, they might well walk straight out and go to the school's nearest competitor. This lack of practicality would mean the test was not "fit" for its purpose as a sales tool.
- if a practice test for an exam such as the Cambridge Flyers test (intended for 7-12 year olds) was administered to a group of teenagers, they might feel it was too "babyish" and be demotivated. The content would not be age-appropriate and therefore not "fit for purpose".
- if a group of learners were taking a Business English course in order to improve their ability to give Sales Presentations, and in the diagnostic test for the course were just asked to chat to the teacher about what they had done at the weekend, the test would only give evidence of their ability to participate in short turns on conversational topics - it would not show not whether they were able to organise and deliver a long turn using lexis specific to their product or service, or persuasive language necessary in a sales context. As well as the problem of lack of evidence of their true communicative needs, it might also lack face validity : they would not see the point of having spent time on something irrelevant to their work, and again, this might result in demotivation at the beginning of the course, while one purpose of a diagnostic test should be to assure the learners that the course will cover exactly what they need/want it to. Both these factors would therefore mean that the test was not "fit for purpose".
- if the same test were given to the learners at the end of the course, it would not only lack content validity (presuming that the course had focused on Sales Presentations) but also predictive validity as it would not produce evidence for the learners or their company as to whether they would now be able to meet the demands of their jobs when speaking to clients. Again - the test is not "fit for purpose"
- if a direct test such as a writing test had no specific set of marking criteria, ensuring that a certain number of marks were awarded to specific areas (eg layout, organisation of ideas, grammatical range and accuracy, lexical range and accuracy, stylistic appropriacy etc) different markers might emphasise some areas over others. This would mean that the marks awarded to any text would depend as much on who by chance marked the test as on the actual content of the text itself. This lack of marking reliability would mean that the test did not accurately discriminate between the learners in the cohort (Learner A might gain 70% if her text were marked by Teacher X, while Learner B gained only 40% because teacher Y was using different criteria) and this would again mean the test was not "fit for purpose".
These are only a few examples, but serve to show how, while all the factors are important for all tests, in specific circumstances some will be more important than others in determining exactly how "fit for purpose" the test is.
An ELT Glossary : Coverage
Coverage is a quality of a test, and refers to the number and appropriacy of the items included. This should be balanced and reflect what the learner has been taught and/or is expected to know. A test might have inadequate coverage if :
a) it was too short overall, so that only a small portion of what needed to be tested could be included. Determining test length therefore has to be a balance between ensuring that the test is long enough to provide adequate coverage, but not so long as to tire / bore /demotivate learners. This is an issue eg with general purpose placement tests, which must contain enough items from each level to discriminate between learners, and ideally would test all four skills as well as systems items such as grammar or lexis. However, as placement tests are frequently used in commercial language schools before enrolment (and are therefore part of the sales stage), they must not be so long as to make the learner decide not to stay for the test.
b) it over-emphasised some items at the expense of others - eg if in an end of course achievement test 70% of the items related to lexis focused on lexis connected with the environment, even though this had only been covered in one out of the ten units of the course, and the test included only 30% of items testing the lexis from other units and the topics covered there.
c) it overemphasised some systems/skills at the expense of others, for example if the test contained only items testing grammar, lexis and reading, while the course had given equal emphasis to the development of listening.
d) it overemphasised some sections of a course at the expense of others - eg a progress test for Units 5-8 of a course, which was made up 80% of items from the final two units and only 20% of items from units 5 and 6.
e) It was too easy or difficult. For example a proficiency test which aimed to demonstrate that learners were at a strong CEFR B2 level, but only included items that would be expected to have been mastered at A2 and B1 with no coverage of items that a B2 level learner would be expected to know.
Coverage is often divided into different types, of which the most commonly used are Content Coverage, Curricular Coverage and Domain Coverage. Click on the link for more precise details, and for the way coverage can affect both validity and reliability.
An ELT Glossary : Transparency
Transparency is a test quality which refers to the extent to which it is clear to the learner what s/he has to do. This may be affected by factors such as :
- The clarity of the instructions, including whether they are written in language that the learner understands at his/her current level.
- If the test type includes individual items - eg a gap-fill activity - whether an example is given after the instructions to illustrate them.
- If the task is more complex, if the learner knows what the marks are awarded for - eg in a writing activity, a specific number of marks may be awarded for each of layout, organisation of ideas, grammatical range and accuracy, lexical range and accuracy, stylistic appropriacy etc.
- If the learner knows how much time is available for the test and, if it consists of more than one task, how much time s/he is recommended to dedicate to each.
- If the learner knows the exact start and finish times of the test. In a classroom setting these may be written on the board, while if the task is taken online the learner needs to know if time is unlimited or whether s/he will lose access after eg 30 minutes.
- How familiar the learner is with the test type. If a preceding course has included a number of activities of the same type, the learner won't have to think about the instructions or examples for long but will "automatically" know what to do.
The last point, however, has two disadvantages. Firstly, it's the reason many exam preparation courses are little more than interminable "ploughing through" past papers, with a greater focus on exam strategies than on actual language improvement. Secondly, if the learner expects a specific format and there is a slight change, s/he may not have read the instructions carefully, not spot the change, and therefore lose marks. An example of this would be eg a test which had always included a task where learners could choose four our of a possible six questions to answer. If the test suddenly changed to only three out of the six, the learners might waste time on a fourth. If it suddenly became obligatory to answer all the items, the learner might again not spot the change and throw away marks by not completing the activity.
An ELT Glossary : Norm and Criterion Referenced Testing
In norm referenced testing, the percentage of the test cohort (ie the people taking the exam) who will pass or get certain grades is decided in advance. Eg if I have a class of 20 learners I might decide in advance that, when the test is marked, the top 25% (ie the 5 learners with the highest marks) will get Grade A, the second 25% Grade B, the third 25% grade C, and the final 25% will fail. This means that if in June I give the test to a very good group, someone with 60/100 marks may well fail - because 15 out of the 20 learners got higher marks than this. However, a learner belonging to a much weaker cohort, which takes the test in December, might gain Grade A with 60/100 - because s/he was in the top 5%.
This unfair discrepancy is avoided by criterion referenced testing, which measures learner performance against a set of predetermined factors - ie descriptions of what learners are expected to know and/or be able to do at a specific stage of their learning.
For example, some simple criteria for a writing test at B2 level might include eg :
- Divides the essay into logically ordered paragraphs, each focusing on a single topic (or two shorter but logically connected topics)
- Uses a topic sentence to alert the reader to the topic of the paragraph.
- Uses connective expressions accurately and appropriately to show the relationship between ideas
Criterion referenced testing means that all learners who meet these standards will pass - regardless of whether this is 100% of the cohort, 0% or anywhere in between. My learners in June and December who score 60/100 will get the same grade, regardless of whether the test is taken in June or December. Learners are judged by what they can do - not by comparison with others in the cohort.
An ELT Glossary : Proficiency Tests
Proficiency tests aim to certify that a learner is at a specific level, regardless of what previous course(s) s/he has or has not done.
Many proficiency tests are run by external examining boards such as Cambridge English. For example, the Cambridge First exam is a proficiency test which, if passed, certifies that the learner is at CEFR B2 level in terms of language proficiency; Delta Module One certifies that the learner has an advanced knowledge of TEFL methodology and an advanced ability to analyses English grammar, lexis, phonology and discourse.
An ELT Glossary : Test Battery
A test battery is a test which involves more than one activity type - in other words, the test is composed of a group of activities which may be scored individually or whose scores may be combined into one overall score. For example, the Cambridge English PET has three components :
- A Reading and Writing test
- A Listening test
- A Speaking test
The results are combined into a single score, which is then graded as Fail, Pass, Pass with Merit, or Pass with Distinction
An ELT Glossary : Impact
Definition : Impact refers to the socio-economic effect the introduction of a test may have.
Example : Since specific IELTS grades have been adopted by a wide range of universities as an entrance requirement, a significant market has arisen for IELTS preparation courses,with a resulting positive economic effect for the language schools which run these courses and/or act as an exam centre.
An ELT Glossary : Formative and Summative Assessment
Assessment of learners may be carried out for either of two purposes :
a) Formative assessment is used to improve the quality of future learning - ie to help the teacher and learners decide how successful their learning has been up to that point, what needs recycling and consolidation before they can move on, whether different learning strategies need to be introduced and used etc etc. Examples of tests with a formative purpose are diagnostic tests and progress tests.
b) Summative assessment, on the other hand, evaluates the success of past learning in terms of pass/fail or various forms of grades. They show to what extent the learner has or hasn't achieved the standards required by the programme. Examples of summative tests include achievement tests and proficiency tests.
An ELT Glossary : Proficiency Tests
Proficiency tests aim to certify that a learner has reached a specific level of competence in the language. For example, if a learner can pass the Cambridge English Preliminary English Test, it shows that s/he has reached CEFR B1 level; if on the other hand s/he can pass the Cambridge English First, that s/he has reached CEFR B2.
Proficiency tests are generally administered by recognised examining bodies (such as Cambridge English) and are often the aim and end point of the course the learner is following.
An ELT Glossary : Progress Tests
Progress tests are tests given during a course to check how well the items/subskills taught up to that point have been assimilated and retained. They have a formative purpose - ie they help the teacher and learners decide how successful the learning has been to that point, what needs recycling and consolidation, whether different learning strategies need to be introduced and used etc etc.
For example, a text book with twenty units might provide a test after every five units which focuses on the language items/subskills that the learners have met in those units.
An ELT Glossary : Placement tests
A placement test is a test used to determine which course a learner should attend. It would be used in eg a private language school to assign newly-arrived learners to a class at an appropriate level.
A placement test will therefore need to cover a range of items at all levels from beginners (CEFR A1) to advanced (CEFR C2). If it contained 60 items therefore, ten might be at level A1, ten at A2, ten at B1, and so on. A learner who scored highly on the first 30 questions, but whose scores dropped off immediately afterwards would therefore be seen to have assimilated the language commonly taught at A1-B1 level, and to need to study at B2 level. S/he would be placed in the class accordingly.
Recommended Reading
Hughes, A. Testing for Language Teachers, CUP
An ELT Glossary : Discrimination (testing)
The term discrimination, as used in testing, refers to the extent to which a test distinguishes between strong and weak students. When a test is pre-tested, attention will be paid to each item's item discrimination index. If all the members of the cohort get that items right, or if all get it wrong, the item will be rejected - it gives no evidence of the differing knowledge/ability of the members of the cohort and is therefore useless for testing purposes. If, on the other hand, only 20% of the pre-test cohort get an item right, and are the same 20% who score highest on the test overall, then the item clearly discriminates between the "top" group and the rest.
Davies (1999) suggests that, in general, an item would be expected to have a difficulty index of 30-70% to be useful in discriminating between candidates. He points out, however, that this will vary depending on test purpose. In a placement test or diagnostic test, difficult items may be included because of the aim of the test - to find out the candidates exact level, or what they do and don't know. For a test such as an achievement test, which expect a high pass rate, items which the candidates are expected to find easy may be included in order to give representative coverage to all the items taught on the course.
Bibliography : Davies, A. (1999) A Dictionary of Language Testing, CUP
An ELT Glossary : Diagnostic Test
Definition : A diagnostic test is a test given at the beginning of a course which aims to discover exactly what the learners know or don't know already and where their strengths and weaknesses lie. This information is then used to decide the course content. Diagnostic tests are therefore formative in function.
Example 1 : A teacher in a private language school has a new group of learners. In the school placement test, they all tested out at CEFR B1 level. However, they have all studied in different contexts previously, and the teacher does not therefore know exactly what they have covered. She therefore designs a diagnostic test which covers :
a) the language items and subskills usually encountered at A2 level. Their performance on this part of the test will tell her which items they have fully assimilated and which will need to be focused on before moving on.
b) a range of items and subskills usually encountered in the early stages of a B1 course and covered in the textbook that she intends to use. Their performance on these items will tell her whether they in fact do know some of the items that she would expect to have to teach from scratch, but which in fact will only need to be recycled and consolidated.
Example 2 : A CEFR C1 level learner is attending a one-to-one tailor made Business English course to improve his ability to negotiate in English. A diagnostic test is designed which focuses on language items and subskills commonly needed in negotiations, and including a negotiations simulation which is filmed and analysed by the teacher. From this, the teacher can identify the learner's weak areas, and design a course to meet his needs.
Further reading
Baxter, A. Evaluating Your Students, Richmond Publishing
An ELT Glossary : Content validity
Content validity is a test quality concerned with whether the test under consideration tests only what has been covered in the preceding course (for progress tests), the preceding course syllabus (for achievement tests) or test specification (for proficiency tests).
Examples :
- in their course, the learners have been studying the present continuous for ongoing present events, These are included in their progress test, but the reading comprehension also includes several examples of the present continuous used to express future arrangements - and tests learners on their understanding of this. This test lacks content validity.
- a professional qualification for language teachers has a methodology syllabus which consists of 100 items. In the written exam which they sit, they are asked about something which is not included in the syllabus. This test lacks content validity.
Related terms : validity, construct validity, face validity, predictive validity
An ELT Glossary : Validity
Validity is a test quality. A test is valid to the extent to which the assessment procedure tests what it is supposed to test. There are, however, a number of different types of validity, and these need to be considered individually. The most important, at least in terms of language learning, are probably:
An ELT Glossary : Predictive Validity
Definition : A test is said to have predictive validity, if the results accurately indicate whether or not the test taker will be able to perform a specified "real life" task outside the teaching/testing context . In language testing this obviously implies "...will be able to perform a specified "real life" task in terms of the level of language competence required".
Example 1 : The IELTS exam is intended to predict whether or not the test taker will be able to cope, as far as English language competence is concerned, when following a university course taught in English.
Example 2 : The Occupational English Test for Healthcare Professionals is intended to predict whether various healthcare professionals (doctors,dentists, physiotherapists, nurses, veterinary surgeons, pharmacists etc) have a sufficient knowledge of general and medical English to work successfully in an English-speaking healthcare context - eg being able to communicate effectively with patients, relatives and colleagues.
Other types of validity include : content validity, construct validity, and face validity.
Other types of validity include : content validity, construct validity, and face validity.
An ELT Glossary : Face validity
Face validity concerns the extent to which a test appears valid - ie appears to be "a good test" to the people using it - the learners taking the test, their parents, the teachers and institutions putting them in for the test etc.
As these people may not be testing experts, they may of course not fully understand what is involved. As an example: if you look at the example given under construct validity, you will see that there are various reasons why dictation is not a valid test of listening comprehension. Yet many learners (and teachers) who are used to it as an activity type may accept quite happily that listening ability should be tested in this way, and make no objection to a test which includes it.
This means that a test which has face validity is not necessarily a good test - and similarly a test which does not have face validity is not necessarily a bad test. However, despite this, it is an important quality for a test to have. If a test, which may possess every other quality required - other forms of validity, reliability, practicality or whatever - does not have face validity, then learners are not going to want to take the test or be happy that its results are a fair assessment of their language competence.
Related terms : validity, content validity, predictive validity.
Subscribe to:
Posts (Atom)