WAITING FOR TEF: WHY IS A TEACHING EXCELLENCE FRAMEWORK ON THE AGENDA AND WHAT MIGHT IT LOOK LIKE?

The announcement by Higher Education Minister Jo Johnson MP on 1st July, that the government will introduce a Teaching Excellence Framework (TEF), should have come as no surprise to the sector. The idea of having a TEF to match the REF (Research Excellence Framework) has been kicking around in the sector for a while. It was discussed at a Guardian Roundtable event back in 2013, and was one of the topics up for discussion at a similar event organised by Times Higher Education last year. The intention to introduce it appeared in the Conservative Manifesto, which outlined the commitment to “introduce a framework to recognise universities offering the highest teaching quality”.

If the broad idea was firmly on the agenda, what a TEF might actually look like in detail and what this might mean for the sector has been subject to considerable discussion and speculation. The Business, Innovation and Skills Select Committee has just launched an inquiry which will examine TEF, and issued a call for evidence to be submitted by the end of this month. We will have to wait until the publication of the Green Paper, promised this autumn to see the details of what the government is proposing, but even at this stage, there is plenty of scope to map out some of the key issues that are coming through the debate in the sector.

Before getting into the details of policy, it is worth setting out some aspects of the context in which the TEF proposal have emerged. Probably the most important of these has been the raising of the maximum level of undergraduate fees to £9,000 in England. Perhaps it was inevitable that this would lead to renewed questions about the value of higher education and the quality of teaching that students were receiving. Such concerns are not new and were one of the areas explored by a Select Committee Inquiry on universities and students in 2009. But they have been fuelled by work such as Academically Adrift, the influential 2011 book by US sociologists Richard Arum and Josipa Roksa. Their study of over two thousand students in the United States suggested that nearly half did not demonstrate statistically significant improvement in their critical and analytical skills during their first two years of study. David Willets, Higher Education Minister during much of the Coalition’s period in government, was influenced by this research, referring to it in his 2013 speech to the HEFCE Conference. Indeed, part of the explanation given by Arum and Roska, draws on George Kuh’s idea of the low engagement contract in which both staff and students settle for lighter teaching and assessment loads to increase the time for other activities (research for the academics, and part-time work or social life for the students). This same idea appeared in a Jo Johnson speech last month in a section dealing with the TEF.

However, the focus has not only been on the choices made by higher education institutions, students and teachers. Writing in 2013, Willets acknowledged that the incentive system put in place by governments in recent decades had created a strong incentive to improve research performance through the REF and Research Assessment Exercises (RAE), but had failed to do the same to promote teaching. Johnson followed a similar line in his 1st July speech, stating “it is striking that while we have a set of measures to reward high quality research, backed by substantial funding (the Research Excellence Framework), there is nothing equivalent to drive up standards in teaching”. It is the government’s intention that this will change with TEF. Success in the TEF will mean that universities will be allowed to raise their tuition fees in line with inflation, and this incentive is scheduled to be in place for the 2017-18 academic year.

So how might TEF do this and what might it look like? Given the tight timescale for implementing TEF, the starting point will have to be the metrics that already exist. Probably the two most well-known of these are the National Student Survey (NSS), which measures student’s satisfaction, and the Destination of Leavers from Higher Education (DLHE) survey, which collects salary and employment data. This data is already publically available at course level through the Key Information Set (KIS) datasets and, with the addition of indicators relating to student progression, may well form the basis of the TEF.

There are, however, significant challenges in using such indicators as measures of teaching quality. NSS is currently focused on measuring student satisfaction rather than student engagement. While research suggests that measures of engagement are good indicators of student learning, evidence for the former is weaker. In addition, NSS results tend to vary by discipline, with students consistently reporting higher levels of satisfaction in some subjects than others. This is not an uncommon feature of course experience questionnaires. As Graham Gibbs has observed it is “highly unlikely” that this is evidence that some subjects are taught better than others, and more likely to reflect biases in the surveys themselves. There is ample evidence that retention and achievement rates not only vary between subjects, but also reflect patterns of social and economic inequality. As the Prime Minister acknowledged in his 2015 Conservative Party Conference speech, inequalities and prejudices influence recruitment processes and these impact upon graduate employment outcomes.

If indicators such as NSS, DLHE and retention data are to be key elements of the TEF, these factors need to be taken into account. Firstly, each indicator can be benchmarked to take account of student characteristics, discipline level differences and other factors such as the region in which a university is based. This could level the playing field between institutions with different profiles and missions in a way that existing league tables fail to do. Secondly, the indicators that are available can be improved. Revision to the NSS are scheduled for 2017 and are expected to include new questions that aim to measure student engagement. HEFCE is also currently undertaking twelve pilot projects to explore how learning gain might be measured. The provisions of the Small Business, Enterprise and Employment Act passed earlier this year should also provide more accurate data on graduate earning and employment.

A second aspect that will determine the shape of a TEF is the question of whether the assessment of teaching excellence is undertaken at course or whole institution level. If TEF performance is recognised at institutional level, this would avoid the potentially confusing situation in which courses within the same university charged different levels of fees. In addition, it would also recognise the contribution of university wide services such as libraries. However, evidence suggests that in many universities, there is significant variation between the performances of different departments. If TEF performance is linked to the fees incurred by students, is it legitimate that they  pay a higher fee level because the institution as a whole is that is rated as excellent, although their own department is not? This would suggest that the alternative approach of course or departmental rating might be preferable and would allow for the extension of the REF principle of funding excellence at a discipline level wherever it exists.

There are a range of ways in which these two alternative approaches could be combined. One option would be to require the institution as a whole to reach a certain threshold level of performance and then allow individual units to be entered to an assessment process. Such an approach has been compared to the process for the Athena Swann gender equality awards, and there has been suggestions that a similar Gold, Silver and Bronze departmental rating system could be developed for the TEF. This approach would allow for the recognition of excellence at the level at which academic staff and student identity tends to sit. It would also allow for a clearer comparison between courses in the same subject at different universities. However, if the link to fees remains at the institutional level, problems of aggregating departmental ratings would remain.

There are unlikely to be any quick fixes to some the issues outlined above, but there is scope for the TEF to take account of the existing differences between institutions, disciplines and students, and evolve to include better indicators over time. If it can be structured to reward those departments which are most successful in supporting student learning and incentives others to emulate their achievements, then TEF may become a valued part of the higher education infrastructure.

Professor John Craig is Head of the School of Social, Psychological and Communication Sciences at Leeds Beckett University and Chairs the PSA Teaching and Learning Specialist Group.

Image: Jisc infoNet: CC BY-NC-ND