Helen Williams 2 December 2020

 

My dual capacities as the Director of Education and Student Experience in my own department, with responsibility for responding to external examiners’ reports, and recent experience as a new external examiner at another institution have led to some reflections about the state of external examining in the UK, with my unease settling on two areas: rigour and inclusion. The PSA webinar on 1 Dec 2020 was aimed at starting a conversation about these issues, with the kind support of the PSA Executive to bring about change.

Rigour

I would adopt the Cambridge dictionary’s second definition of rigour – ‘the quality of being detailed, careful, and complete’ – rather than the first definition (unpleasant or severe rule-following) for what I mean by rigour in external examining. The UK Quality Assurance Agency lays out some very clear expectations about the role of external examiners in ‘Chapter B7: External Examining’ of the UK Quality Code for Higher Education. In indicators 1-3, they establish the key responsibilities of externals:

Provide informative comment and recommendations on whether…

  • Academic standards are maintained, including meeting applicable subject benchmark statements;
  • Assessments are rigorous, fair and assess the programme outcomes;
  • Standards are comparable with other institutions.

Share good practice and recommendations for innovation and learning enhancement.

Elsewhere in the document, the QAA provide greater detail about these tasks, including:

  • Provide critical feedback and suggestions for improvement.
  • Check appropriateness and consistency of marking schemes/criteria.
  • Check assessments across modules are of comparable standard.
  • Check for alignment between learning outcomes and QAA subject benchmarks.
  • Ensure the curriculum remains current.
     

The points above are ranked from highest to lowest agreement amongst the webinar participants of how much they thought externals were performing these tasks for their department. The greatest agreement was that externals provide critical feedback and suggestions for improvement (mean 4.4/5; with 1 meaning ‘never’ and 5 meaning ‘always’). This is perhaps unsurprising, given that academics are well trained in criticality, and externals are often prompted by the forms used to offer feedback and suggestions for improvement. Most participants also felt that externals check assessments across modules (mean 4.2) and check appropriateness and consistency of marking schemes/criteria (mean 4.2). This is probably again a reflection of how often we do these tasks within our own departments as part of our internal moderation processes; they are therefore more likely to be considered routine by external examiners and appear to be widely understood as core to the job.

 

On the other hand, the remaining two areas were much less common, with ‘never’ being the most common response for checking alignment with QAA subject benchmarks (mean 2.8), and the lowest score for ensuring the curriculum remains current (mean 2.6). This aligns with my experience in responding to examiners’ reports: mentions of the subject benchmarks are rare, and while points about the curriculum are more common in my own experience, they are still far less frequent than comments about module assessment and marking criteria.

No matter how high-quality the comments from externals, however, the process would not be rigorous if the comments did not have a positive impact on the quality of provision. In the QAA guidance, indicators 14-16 state that:

  • The department and institution should ‘give full and serious consideration to the comments and recommendations’.
  • Reports should be available in full to students.
  • Student reps should be involved in the process.
     

Anecdotal evidence suggests that institutional processes for consideration of comments and implementation of changes are becoming more robust and widespread, but the seriousness with which reports are considered varies substantially between institutions, just as the volume and detail of comments varies considerably between examiners. Perhaps the only consistent point between institutions and examiners is acknowledgement that there is no accepted standard of processes and expectations.

Identifying this variability is hardly a new point. The Dearing report (1997, point 10.94, p. 162) specifically stated:

“The remit of the external examiner will need to be consistent across the UK, necessitating thorough familiarisation, training and preparation, including a trainee/apprentice model for new external examiners. Examiners will need to be fully aware of the aims, teaching methods and approach of programmes under examination.”

Discrepancies in expectations were highlighted again in a report commissioned by the Higher Education Academy (now part of Advance HE) and the QAA in 2012. This research worked with experienced external examiners across a range of different subjects and found:

  • ‘Few clear patterns emerged’ (p. 5);
  • ‘Where similar constructs were shared, they were interpreted differently, appearing to result in manifestly different standards’ (pp. 5-6);
  • ‘Analysis of [the overall marks externals assigned to student pieces] revealed little inter-examiner agreement’ (p. 6);
  • ‘A number of often contrasting standpoints with regard to what external examining entails and what standards should be used in examining processes’ (p. 6).
     

Advance HE has attempted to address this by creating a professional development course for external examiners and maintaining a directory of examiners who have completed the course. This is laudable in its aims at increasing consistency and rigour, but it is offered inconsistently, oversubscribed, costs more than most externals’ annual fee (before tax) and requires 20-25 hours of study over 40 days – all of which represent substantial barriers to inclusion and set such a high bar that few externals are likely to meet it. This leads me to the other main point about the state of external examining…

 

Inclusion

The HEA’s own External Examiners’ Handbook acknowledges:

“Institutions rarely advertise in the press for external examiners and the knowledge of availability of posts may still rely heavily on word of mouth and networks within a discipline area…[T]he most effective way of becoming an external examiner is through networking with discipline colleagues at conferences and meetings or through links developed in research and teaching.”

This runs counter to a standard pillar of good practice in equality, diversity and inclusion that positions should be advertised and subject to a transparent, fair recruitment process. The primary problem here appears to be driven by a lack of coordination and infrastructure: there is no clear jobs board where external examining posts are advertised, which leaves universities reliant on personal networks to identify suitable candidates.

I was struck by the overwhelmingly white, male appearance of our own panel of external examiners in a recent exam board, and a straw poll of the webinar participants demonstrated that this lack of diversity is reproduced at many other institutions. In fact, when we recently appointed a new external examiner, we tried to diversify our panel, but lacking a clear pool of people who were keen to be considered, we were not successful in this aim.

I finish with a call to action from these observations to encourage our discipline to take this forward and ensure accessible, affordable training of examiners as well as a much more transparent appointment process that allows prospective candidates to identify themselves and apply for positions rather than waiting for a tap on the shoulder or spreading the word through personal networks. I feel the professional associations for our discipline in the UK (PSA and BISA) are best-placed to coordinate these actions and support the PSA in trying to create a solution to this.

Further Reading

Dr Helen Williams is an Associate professor of politics at the University of Nottingham. Her research areas are citizenship, immigration, stats, decolonisation, teaching. To watch Helen's full webinar on this subject click here.