Scottish independence: Are the referendum polls wrong?

  • Published

Ten days ago, one opinion poll suggested for the first time that the Yes Scotland campaign had a narrow lead.

Each poll since has been close, and has been hailed in the media as showing that the future of the United Kingdom is on a "knife-edge".

Some, though, believe the polls driving the headlines could be wildly inaccurate. Martin Boon, director of polling company ICM, warned BBC Radio 4's 'More or Less' that the independence referendum could prove to be a "polling Waterloo".

So, could the polls be getting it wrong?

What are the polls saying?

Image source, WhatScotlandThinks

The reason you have heard so much about the opinion polls recently is because the gap between 'Yes' and 'No' has apparently dramatically narrowed.

The latest "poll of polls" - published on the What Scotland Thinks website - shows the results of the last six polls combined. Once undecided voters are excluded, it puts the 'Yes' vote on 49%, while 'No' is just ahead on 51%.

Contrast that two percentage point gap with 1 June, when 'Yes' was on 42% and 'No' was on 58%.

The experts agree: there has been an increase in 'Yes' support in recent weeks and the polls now appear to be very, very tight.

How are the polling companies getting these figures?

Image source, Thinkstock

Each polling company has its own methodology: they decide what questions to ask; how many people will be asked them; and whether interviews are conducted face to face, over the phone, or online.

Mr Boon believes whether people are polled online or by phone can affect the outcome.

Phone surveys, which typically go through landlines, have trouble reaching young people and there tends to be a very low response rate.

With online polls, the panel has chosen to take part, meaning they are probably more interested in polling and politics than your average punter, and this same pot of people are likely to be used by multiple companies again and again.

However, some experts believe people tend to be more honest online due to a higher degree of anonymity - there's no interviewer there to judge them.

It is important to note the independence referendum is the first major political event in the United Kingdom in which the vast majority of polls have been conducted online.

Why could they be wrong?

Image source, Thinkstock

Any poll of 1,000 people has a margin of error of 3%, but a specific problem with opinion polls for the referendum is that there is no precedent to which voting intentions can be compared.

Mr Boon explained: "If I could peg my data back to a previous Scottish independence referendum, life would be good. But I can't - and this is one of the black holes which has made the life of pollsters really rather difficult."

On top of that, there is a particular problem with online polls as the polling company needs to apply special weighting to different respondents to ensure the results are as representative as possible. That is a complex process.

Anthony Wells, director of YouGov, said: "In some ways, (online samples) can't be representative. For example, little old ladies who are on the internet are more likely to be affluent which we can account for, but what we can't control are (variables) such as them being more likely to be more comfortable with technology."

Essentially, samples need lots of tweaking to make them more representative, and a lack of historical precedent to draw on, along with problems unique to online polling, makes this process very, very difficult.

How wrong could they be?

Image source, Thinkstock

Very. Mr Boon said all polling companies are relying on a "relatively small pot of Scots" which is "not big enough" and "a real concern".

"This referendum has the potential to be a polling Waterloo", he said.

"I think the best we can hope for as an industry is that we dodge a bullet, but it's entirely possible that the bullets do start spraying our way."

Some have argued that there's a "missing million": a reawakened section of the electorate who have either never voted before, or became disillusioned with politics long ago.

The 'Yes' campaign believes their polling figure is underestimated because they are picking up support from this section of society in working class areas, where people generally haven't registered to vote before, don't have a landline for phone polls, or don't choose to take part in online polls.

If they're right, it is possible the true figure of the 'Yes' side is higher than pollsters think.

Have they got it wrong before?

In 12 out of 16 recent referendums, the final poll figure for 'Yes' was higher than the actual results, according to research by political sociologist Dr Stephen Fisher.

Dr Fisher explains: "The experience of referendum polls in the UK and internationally suggests that the findings of final polls are typically flattering for the 'Yes' camp."

The episode which still sends shudders through pollsters is the 1992 general election, when even the BBC's exit poll suggested Labour had narrowly won. In fact, John Major went on to form another Conservative government.

One of the key phenomena here became known as the "spiral of silence": the Tory vote was underestimated by polls because people either did not want to be seen favouring an "unfashionable" option; or initially considered voting Labour but went for the less risky choice of the Conservatives at the last moment.

Dr Fisher believes that, in the Scottish referendum, this could mean the 'Yes' polling figures are unrepresentatively high.

So, if the polls are wrong, what's the point in polling?

Image source, Thinkstock

Many countries ban polling for at least part of political campaigns.

But, before you dismiss political polling entirely, it's important to say that polls are generally reliable when there is a precedent.

Even in the referendum, where the polls are possibly on less stable ground, the data has undoubtedly shown a "real, sustained, and large change", says Anthony Wells.

He said: "There are a number of real individual people in our sample who have moved towards Yes, while six months ago they were 'No' or undecided."

So the polls have identified that the momentum is from 'Yes' or 'No', but there is more doubt about the absolute levels of where that support stands - and, if the vote really is close, that is key.

All will become clear sometime in the early hours of Friday morning.