Stephen Fisher, Martha Kirby and Joe Greenwood

 

Executive Summary

 

The U.K. Political Studies Association today released the results of its survey of expert predictions for the 2019 general election. The survey was distributed to members of the Association as well as survey researchers from major polling companies in Britain and to journalists from the print and broadcast media. Fieldwork was conducted between 20th November and 2nd December 2019, with most responses in the first few days of that period. At that point the opinion polls were recording Conservative leads over Labour of 13 points on average, and forecasting models were suggesting that the Conservatives were on course for a majority of around 60.

 

We asked respondents to share their predictions of the vote shares for each party, the number of seats won, the probability of a Conservative majority of more than 100 seats, and the level of turnout.

 

• Half of the respondents forecast a Conservative majority while half predicted that the Tories would fall short. On average they gave a 19% probability of Boris Johnson winning a majority of more than one hundred seats.

 

• The average vote share prediction for the Conservatives was 39%, with a lead of eight points over Labour. This is below the average leads in the polls at the beginning (13 points) and end (10 points) of the fieldwork period. The predicted share for the Liberal Democrats was 15%, with 6% for the Brexit Party and 4% for the Greens.

 

• There is a prevailing view among experts that the Conservatives will end up as the largest party in parliament: the median prediction was that the Conservatives would win 326 seats, Labour 231 seats, the Lib Dems 23 seats and the SNP 45.

 

• The average predicted turnout for the election was 66%, slightly below the 69% turnout at the general election in June 2017 and far below the 72% achieved at the 2016 EU referendum.

 

This survey is the fourth expert forecasting exercise that the PSA has run. All three previous occasions were ones in which the character of the result came as a surprise to many. Experts typically predicted a hung parliament in 2015, a Remain win in 2016, and a Conservative majority in 2017. To that extent the expectations of experts were broadly in line with most opinion polls and forecasting models at the time. Perhaps most striking this year is that, despite having been more optimistic for the Tories in 2017 than most other predictions, the experts are now somewhat more cautious about Conservative prospects than betting markets, and nearly all polls and forecasting models. This and all the other findings in this report should be interpreted in light of previous experience.

 

 

Stephen Fisher, Associate Professor of Political Sociology at Trinity College, Oxford observed:

 

“The Political Studies Association expert survey findings concur with general expectations of a Conservative victory at this election, however the expert respondents were divided over to whether they think Boris Johnson will win the majority in parliament that he needs to deliver his Brexit deal.”

 

Joe Greenwood, Fellow at the London School of Economics and Political Science (LSE), said:

 

“In the judgement of the experts who completed the survey, the wide gap between the Conservatives and Labour in the opinion polls is likely to close somewhat. The experts anticipate a tighter result on polling day than would be the case if the public voted today and seem to be anticipating a rerun of 2017, albeit a less dramatic version.”

 

Martha Kirby, Doctoral Student in Sociology at Nuffield College, Oxford remarked:

 

“Whilst the forecast vote shares from the 2019 PSA Expert Survey are broadly in line with vote-intention opinion polls, the experts in the survey are much more sceptical than most forecasting models as to whether the Conservative lead on votes will translate into a majority of seats for the party. This fits with the predominant media narrative of being cautious about reading too much into the polls.”

 

The following are key tables for the report:

 

 

 

 

The survey elicited responses from 380 academics, pollsters, and journalists. Of these 191 gave forecasts for headline votes or seats for the two main parties. The fieldwork was conducted between 20th November and 2nd December 2019.

 

 

 

Introduction

 

On December 12, 2019 the UK will have its third general election within five years. Together with the referendums on Scottish independence and the UK’s membership of the EU, this will be the fifth major electoral event in the last six years. The outcomes of these events have shocked many.

 

Ideally though, they would not have come as a surprise. Not because they should have been foregone conclusions, but because public opinion and electoral behaviour ought to be well measured in opinion polls, well understood by analysts and somewhat predictable, at least close to the event once the major policies and arguments have been aired. True, there is no anticipating campaign events, and some of the campaigns in recent years did include momentous events. But ultimately the surprise outcomes were primarily due to the failure of the majority of polls and forecasts to point to salient features of the eventual outcome.

 

Some researchers argue that forecasting political outcomes is a skill that can be learned, through practise. It is partly in this spirit, that on behalf of the U.K. Political Studies Association (PSA) we have conducted a survey of expert predictions of the 2019 general election. We are seeking to understand what expert expectations are like, how they relate to other forecasts, and how the character of expert predictions have changed in light of previous experiences.

 

The survey was distributed online to members of the Association, survey researchers from major polling companies in Britain, and journalists from the print and broadcast media. Fieldwork was conducted between 20th November and 2nd December 2019, with most responses in the first few days of that period. Participation was by invitation only. Responses were anonymous. This was not a competition between individual predictions but an attempt to identify and understand collective expectations.

 

Previous PSA expert prediction surveys were conducted by Will Jennings and Chris Hanretty ahead of the 2015 general election[1], and together with Stephen Fisher ahead of the 2016 referendum[2] and 2017 general election[3]. This report draws heavily on their work.

 

In the report that follows, we describe some of the technical details of the survey and then review the headline results. We are very grateful to all those who took the time to respond and made the not inconsiderable effort of predicting 27 different aspects of the outcome of the elections.

 

 

Response Rate

 

 

Our 2019 expert survey was completed by 380 individuals in total. This is a slight increase from the 335 who took part in 2017, yet still below the number of participants in the 2015 and 2016 surveys. With 2,297 individuals invited to take part in the survey, this equates to a response rate of 17%, only slightly greater than the 15% achieved in 2017.

 

Individuals were initially contacted to take part on the 20th November, with a follow-up email also sent after seven days to try and maximise participation. 79% of responses were recorded within the first three days, between 20th – 23rd November, with the number of individuals choosing to take part after this point falling over time. The low response rate might partially be attributed to the failures of prior surveys to accurately predict election outcomes.

 

Table 1 reports the breakdown of respondents by their respective roles, whilst also highlighting the number of individuals who provided predictions of Conservative and Labour vote share and/or the number of seats in Parliament. In line with previous expert surveys, academic responses constitute the largest proportion, making up 78% of the sample. We are glad that twice as many journalists took part than did in 2017.  This report includes some analysis on variation in the forecasts between academics and others, but due to the far smaller number of pollster responses, attempting to draw comparison between pollsters and journalists would be problematic.

 

 

All respondents who gave forecasts for vote shares in Great Britain provided answers for all major parties. Of those who forecasted seat totals, all gave Conservative and Labour predictions, and the majority also for the other main parties. Non-response rates were higher for parties of Scotland, Wales and particularly Northern Ireland. Questions on those parts of the UK were flagged as optional in the hope that it would be primarily those with an interest in an area that would chose to respond.

 

Before providing their predictions, respondents were first asked to self-assess their forecasting ability relative to their peers, with the breakdown of such highlighted in Table 2. As in 2017, a plurality of respondents assessed themselves as being in the second highest quarter for their forecasting ability. More generally the distribution of self-assessed ability was similar to that in 2017 with most people placing themselves just above or just below the middle.

 

 

 

Those who rated themselves of higher ability were more likely to subsequently provide forecasts of vote shares and/or seats, although this divergence is not witnessed between those who rated themselves as being in the second lowest quarter compared to the second highest quarter. It is unclear whether the variance in completion rate is in consequence of being asked this question. As a result of having more respondents with higher self-assessments and those respondents having higher completion rates, the majority of forecasts come from those with higher self-assessed abilities. However, we should note that in 2017 the accuracy of the forecasts was not systematically related to self-assessed forecasting ability (discussed in more detail below).

 

Context

 

As polls are a snapshot of public opinion at the time, so these expert forecasts are a reflection of the state of play preceding the three days over which the bulk of responses were received. Of course, the experts have to exercise their judgement regarding the likely changes in vote share and projected seats over the subsequent days and weeks of the campaign, but it is nonetheless useful to consider the context in which the answers were provided.

 

The 20th to the 23rd of November, when most respondents gave their answers, encompassed the launch of Labour’s manifesto but preceded the launch of the Conservative’s equivalent document. This period followed the ITV general election debate in which Boris Johnson and Jeremy Corbyn went head-to-head, which neither won according to the post-debate poll. The survey period included Jeremy Corbyn’s poorly-received interview with the BBC’s Andrew Neil, and the London Bridge attacks on the 29th of November, though these events occurred after the vast majority had responded.

 

The polls conducted in the week before 20th November on average had the following vote intention figures for Great Britain: Conservatives 43%, Labour 29%, Liberal Democrats 14%, Brexit 5%, and Greens 3%. In the week preceding our survey the average of polls had continued to show a slow tick upwards for the two largest parties, with the Conservative lead over Labour rising to 13 points by 20th November, up from 10 points the week before. Polls for Scotland, Wales and Northern Ireland are much less frequent. For most of our respondents the most recent poll from each of these regions was several weeks old from the end of October. There was no further poll from Northern Ireland during the survey. Polls from Scotland and Wales during the survey did not show much change for the nationalist parties.

 

Similarly, whilst the bulk of responses had been received by 27th November, when YouGov’s MRP projections were published, the headline figures from that model did not differ substantially from those of published forecasts just before the survey started. With a projection of 359 seats for the Conservatives, the YouGov MRP model suggested a similar majority to the average forecast of 354 seats from sources available on 20th November at the start of the survey. However, given the accuracy of the YouGov MRP model in 2017, the release of the 2019 model may have impacted on answers given in the closing days of the survey response period. Also, the fact that the YouGov MRP projections had not been released in time for most of our respondents, may have contributed to their caution regarding a Conservative majority.

 

Lastly, the broad consensus in the betting markets also pointed to a likely Conservative win, with the odds of a Conservative majority standing at 65% at on 20th November (with  no overall majority on 35%). Those odds did not change much during the survey period, but events may yet change political circumstances from the context in which the experts responded to our survey.

 

Predictions of vote share

 

Respondents were asked to give their predictions of vote share percentages for each of the main parties standing for election in Britain. In Table 3 below, we report the average (mean), median (middle value), 10th and 90th percentiles, alongside the number of respondents.

 

 

 

Overall, 93% of our respondents predict a Conservative lead, whilst only 5% expect a Labour lead and 2% a tie. However, the Conservatives are expected to see a decrease in their vote share, with only 9% of respondents predicting an improvement on their 2017 election result. Despite this, Conservatives are expected to increase their lead because Labour are expected to suffer a substantial fall in their vote share of 10 points on average.

 

Expert-forecast vote shares are generally aligned with opinion polls in the week prior to the survey, as indicated by the final column in Table 3. However, our respondents are more modest in their predictions of Conservative vote share, whilst also being slightly more optimistic for Labour. This might be in response to the 2017 polls overestimating the Conservative lead and substantially underestimating Labour.

 

Our respondents have similar expectations to the opinion polls regarding the scale of the rise in the Liberal Democrat vote share. Fully 98% of the experts predict an improvement on the party’s 2017 result. Following their fall by 2.1 points in 2017, the Greens are also predicted to see a rise in support this year, with expects predicting a slightly better performance than in the opinion polls. As a new contender, the Brexit Party is predicted a vote share of 6%, one point above what the polls in the week prior to the survey were suggesting.

 

Scotland

 

Table 4 presents summary statistics of the predicted vote share for the SNP in Scotland. On average, respondents predict the SNP to perform better than in 2017. Our experts predict a far greater vote share than polling estimates, suggesting a potential return to the 50% vote share attained in 2015.

 

 

Wales

 

Table 5 presents summary statistics of the predicted vote share for Plaid Cymru in Wales. Our respondents predict an average vote share of 22%, which is at odds with the single opinion poll prediction of 12%. Findings show a large amount of variability in answers, evidenced by comparison of 10th and 90th percentiles.

 

 

Northern Ireland

The final set of vote share predictions which respondents gave were for Northern Irish parties, with results of such presented in Table 6. The findings of the most recent poll prior to the expert survey is shown in the final column. The expert predictions fit a similar pattern. The DUP are expected to suffer a decline in the vote share, but still win the most votes overall. Both the SDLP and the Alliance are expected to win more votes than they did in 2017, whilst little change is expected for the UUP and Sinn Féin.

 

 

Predictions of seats in Parliament

 

Our second set of questions related to how many seats respondents thought each party would win in parliament. In Table 7 we report the mean, median, 10th and 90th percentiles and the number of respondents (N) for each party. For comparison, the table also includes the 2017 seat totals and a uniform change projection from the average of the polls immediately prior to the survey.

 

 

Overall respondents clearly expect the Conservatives to win this election, but they are divided on the crucial political question of whether the Conservatives will win an overall majority. Almost exactly half of respondents said the party would achieve 326 seats or more, with slightly fewer than half saying that the Tories would not. The mean forecast was 324 seats.

 

Most of the Conservative’s gains since 2017 would appear to be coming at the expense of Labour, who are forecast to lose around thirty seats. The Liberal Democrats are expected to double their 2017 seat tally from 12 to 25, while the SNP are forecast to increase theirs from 35 to 42. Little change is expected for smaller parties, although the SDLP and UUP are expected to win a seat each even though they failed to do so in 2017.

 

It should be noted that upon inspection of the data, several of the responses for predicted number of seats were erroneous, with the total number of predicted seats in some cases being in the thousands. Therefore, to ensure data quality, we only included responses for those in which their inputted number of seats totalled under 700. This number was somewhat arbitrary and chosen to allow for some margin of error as a result of miscalculations. Subsequent tests also indicated that this had negligible effect on the results in comparison to lower thresholds.

 

Predicted probabilities

 

We also asked respondents to indicate what probability they would put on the events of a Conservative majority of 100+ and 150+ seats. These questions were a holdover from the 2017 version of the survey and, given the polling context at the time the survey was fielded, it would have been better to ask respondents about the possibility of a simple Conservative majority rather than a landslide. This was an oversight which we regret. Nevertheless, it is informative to review the experts’ estimates of the predicted probability of a Conservative landslide, which reflect their cautious estimates for the Conservative vote share and number of seats. The summary statistics for the responses to these questions are reported in Table 8 below. The experts are willing to entertain the possibility of a large Conservative victory but, with a mean predicted probability of a 100+ seat majority for the party standing at 19%, and of 150+ seats at 8%, they are clearly sceptical of this eventuality coming to pass.

 

 

Predictions of turnout

 

As a final question, respondents were also asked to give their predictions of what they think the turnout will be on the 12th December, summarised in Table 9 below. During the 2017 election campaign, there was a great deal of buzz around there being a surge in young voters, the majority of whom supporters of Labour. This year, the media has reported a similar phenomenon, with more young voters registering to vote. Despite this, our experts predict a turnout of 66%, lower than that of the last election (69%), and even further from the EU referendum turnout (72%).

 

Self-assessed Ability of Respondents

 

Do respondents who rate their own forecasting ability more highly provide more accurate forecasts? In addressing this question we look to the results of the 2017 expert survey. Table 10 presents the breakdown of 2017 mean predicted vote shares by forecasting ability, in comparison to the actual election outcome. In 2017, there were only slight differences between groups in their predicted vote shares, with those placing themselves in higher quarters more likely to predict a greater Conservative vote share. All groups were substantially off in their predictions of the Labour vote, however, it was in fact those in the lowest quarter who predicted a higher vote share and were therefore closest to the result, but only marginally and not statistically significantly so.

 

 

 

Table 11 presents the breakdown of 2017 mean predicted seats by self-assessed forecasting ability, in comparison to the actual election outcome. Strikingly, those of higher self-reported ability made less accurate predictions than those of lower ability, overestimating the number of Conservative seats whilst also underestimating the number of Labour seats to a greater extent. The trend is not systematic nor statistically significant but the differences between the top and bottom groups are.

 

 

Turning to this year’s expert forecasts, Tables 12 and 13 show the breakdown of 2019 predicted vote shares and seats respectively by self-assessed forecasting ability. There is practically no difference between the different self-assessed ability groups on their expectations for vote shares.

 

 

On seat forecasts, just as in 2017, those who placed themselves in the top quarter for forecasting ability made seat tally predictions that are better for the Conservatives and worse for Labour than those made by people who placed themselves in the bottom quarter. However, whereas the difference in Conservative seat forecasts between those two groups was 26 seats in 2017, the difference this year is just eight. What is more, there is again no systematic trend whereby the forecast number of Tory seats increases with self-assessed ability.

 

 

It remains to be seen whether those who rated themselves in the bottom quarter for forecasting ability have again provided the most accurate predictions for the two main party seat totals.

 

 

Academic and Non-Academic Respondents

 

Whilst there are not enough pollsters or journalists amongst our respondents to consider those groups specifically, we can compare our academic respondents to our non-academic respondents. Academics gave lower forecasts for the Conservatives on both votes and seats, but the differences are not statistically significant. Otherwise, the forecasts for the two groups are very similar as shown in Tables 14 and 15.

 

 

 

 

Conclusion

 

Overall, the expert forecasts are most distinctive in their more pessimistic outlook for the Conservatives. While the betting markets and nearly all opinion polls and forecasting models point to a comfortable Tory majority, half of the experts who responded to our survey expect the Conservatives to fall short. This is also remarkable given that experts were more bullish than betting markets, polls and forecasters for Tory prospects in a similar survey in 2017.

 

The track record of the expert survey does not suggest that readers should adjust their personal expectations of the outcomes strongly to match the average expert. However, it may well be that the experts have wisely learned to be more cautious about opinion polls and forecasts that rely on them.

 

Acknowledgements: we are immensely grateful to all participants for giving up their time to complete the survey. We hope the results are of interest to them all. Thanks also to Will Jennings and Chris Hanretty, who ran previous surveys and from whose report on the 2017 we draw heavily. We are very grateful to Feargal Cochrane, Michelle Doyle Wildman, Andrew Mycock, John Pollock, Angelia Wilson and especially Lali Sindi at the PSA for supporting, helping set up and distributing the online survey.