The luck factor

Warning: long and boring and technical entry coming up… Feel free to skip it…

I want to talk about the way candidates are marked in their A-level English Oral exams.  No I’m not complaining, but I want to see if any of you has anything to say about the system.  Ok here’s the story:

It’s the Oral exams from 13 Mar to 22 Mar – that’s 8 days of examination.  I’m lucky enough to be one of the examiners.

Each day, I examine 7 groups of candidates (4 in each group) – that’s 28 of them (sometimes 27 or less).  After they’ve given their oral presentation and a discussion, I give them a score from 1 to 7 (1 being the worst, 7 being the best).  Now the score is subjective and based on impression marking; there’s neither criteria nor benchmark nor performance standards (actually there ARE guidelines but I won’t go into that).  I simply compare all 28 of them and sort of rank them.  I’m supposed to fit their scores into something like a normal distribution curve, namely the following "acceptable range" of marks:

1 mark: 0-2 candidates
2 marks: 3-4 candidates
3 marks: 5-6 candidates
4 marks: 8-9 candidates
5 marks: 5-6 candidates
6 marks: 3-4 candidates
7 marks: 0-2 candidates

For example, the following pattern would be acceptable:

1 XX
2 XXX
3 XXXXX
4 XXXXXXX
5 XXXXXX
6 XXXX
7 X

The following pattern would obviously be UNacceptable, for it’s statistically unlikely that the numbers of poor, average and outstanding candidates are all equal:

1 XXXX
2 XXXX
3 XXXX
4 XXXX
5 XXXX
6 XXXX
7 XXXX

The following pattern is also UNacceptable because it’s skewed to one side, indicating that the examiner is unduly harsh in his marking:

1 XX
2 XXXXX
3 XXXXXXX
4 XXXXXXXXX
5 XXXX
6 X
7

Now the obvious question is: How is it possible to make all candidates from EVERY DAY fit in the same normal curve?  I mean, is it possible that the overall standard of candidates on, say, Monday is higher than that on Wednesday?  That is, a certain student getting a 2 on Monday would get a 3 on Wednesday, simply because he’s being compared to a different, less able, batch.  Is this likely?  The Examinations Authority says that it’s theoretically possible but very rare in practice because the candidature of each exam centre is completely random.  Now I don’t know the exact odds for this (I’m no statistician), but this theoretical possibility direct results in a matter of pass or fail.  Why?  Because there has to be a dividing line between pass or fail.  Something like: you get a 2, you fail; you get a 3, you pass.  I know it’s a double-examiner system but that doesn’t change the essence.  Both examiners give a 2; the candidate fails.  Both give a 3; he passes.  You can see that the frequent claim "examiners are subjective" no longer applies here.  It doesn’t make a difference really.

So that leaves the question: Is it likely that a candidate who failed on one day would have passed on a different day?  Absolutely.  In fact, that’s what I experienced in just these 3 days.  On Monday (13 Mar), the standard of my 7 groups was indeed quite high (my partner agreed so).  Today (15 Mar), the standard was unexpectedly poor (echoed by my partner again).  In other words, if you happened to be one of the candidates in my group on Monday, then I feel sorry for you; but if you were in my group today, then congratulations, you would’ve got a high mark!

I know that the Oral component accounts for only 18% of the total Use of English scores, and that it’s again statistically insignificant for a person to fail the whole subject because of this, but the appearance of an "F" on the result slip is still unsightly.

So, what’s the solution to the problem?  Honestly I don’t know.  Can we increase the number of candidates from 28 to 40, to make the normal distribution more likely to be attained?  No, because no examiner can examine 10 groups of candidates in one go and still remain sane or reliable.  Can we change the scoring system altogether and make it a criterion-referenced one, like the LPAT (±Ð®v»y¤å°ò·Ç¸Õ)?  No, because fitting a candidate’s performance to a given level (based on some kind of descriptors) is very subjective and imprecise.  Actually this was done in the past but it was found that examiners often could not discriminate betwee the less able candidates, i.e. the boundary between Pass and Fail was a very fine line to tread.

So, the current system is supposed to be the best one there is.  Or is it?  I really have no idea.

All I know is that when I wish my students "good luck on the exams", I now mean it literally, especially on the Oral exams.