Same Polls, 10-Mandate Gap: Why Samples Paint Different Pictures

Millions of Israelis are expected to participate in the elections and decide who will be the members of the Knesset and the next government. But those who are glued to the screens in an attempt to know the result now must settle for answers based on a few hundred to about 1,000 citizens only. These people, chosen from the entire population using various methods, become in the polls representatives of millions of voters - and provide the data upon which headlines, interpretations, and political forecasts are built.

The point is that the polls do not ask the same people. Each institute builds its own sample, tries to reach different population segments, and uses different methods to turn a small group into a picture that represents Israeli society as a whole. The result obtained is that in the same week, different institutes may present completely different pictures of the mandate map.

So to what extent do the polls actually succeed in representing the millions of Israelis who will vote at the ballot box in about three months, and can the differences between the groups sampled explain the large gaps between the results presented by the different channels?

Emphasis on proportions

Traditionally, on channels 12, 13, and 11, as opposed to channels 14 and i24NEWS, a difference is evident in the final results of each poll. Thus, in recent weeks, while for the former the Likud moves around 21-24 mandates, and the Netanyahu bloc lags significantly - for the latter, the Likud reaches 27-35 mandates, and the bloc leads comfortably.

According to experts, such a gap - of ten mandates or more between channels that are supposedly surveying the same public at about the same time - stems, among other things, from the sample structure of each channel.

Dr. Menachem Lazar, director of the Lazar Research Institute and the pollster for "Maariv", explains that first of all it is important to separate two separate stages of building the sample - the sampling itself and the collection method. "Research institutes examine a group of 500 to 1,000 people, which will be composed of a mix that represents the general population of the State of Israel. When it comes to elections, the entire population from age 18 and up, Jews and Arabs," he explains. "It is necessary that in the main characteristics - ratio of men and women, age groups, districts, etc. - there will be coordination in the sample to the proportions in the population. Pollsters also sample according to the level of religiosity, for example, one of the most decisive variables in the voting behavior of Israelis, so they need a percentage of secular, traditional, religious, and ultra-Orthodox that matches the situation in the country."

Therefore, the important question is not how many people answered, but how accurately they reflect the entire public they seek to examine.

The public that does not answer

To this is added the collection method itself. A telephone survey reaches people in one way, an internet survey or panel reaches in a completely different way, and each such method misses a different audience. An internet-based survey is convenient and fast, but may miss those who are less active digitally. A telephone survey reaches a wider audience, but deals with low response rates - many simply do not answer a call from an unknown number.

Internet panels, for example, tend to examine younger, more educated, and more secular populations - people who feel comfortable with technology and are willing to fill out online questionnaires. And who is left out? Elderly people for whom the internet is foreign, significant parts of the ultra-Orthodox society, some of whom do not even own a smartphone with access to an open internet. Thus, Shas, for example, almost traditionally receives more mandates on election day than in the polls.

The Arab public also poses a challenge: language gaps, suspicion towards unknown polls, and lower response rates to the phone often lead to under-representation or the need to "correct" the results in a statistical weighting, which guesses how much will compensate for the lack.

Mano Geva, who heads the Midgam Institute that conducts the polls for Channel 12 News, explains how to deal with this challenge. "It is possible to sample in a combined way, meaning some of the data can be collected by phone, through a dedicated call center, and some other part on the internet, through an internet panel," he says. "Each platform must suit the population that has access to it. The ultra-Orthodox society, the Arab society, and the older population in the country - are less accessible to the internet, and it is better to interview them by phone. Only in this way is it possible to reach all parts of the country."

Dr. Lazar relies more on an internet panel and explains why this is not necessarily problematic. "An internet panel includes communities of tens of thousands of people who register voluntarily to participate in polls, and provide upon joining their demographic characteristics - age, gender, area of residence, income, education, and religiosity," he says. "I know how to sample almost exclusively through the panel I have been working with for years, and know how to cover the gaps I have in certain populations."

Regarding the fear that those who do not have access to the internet are left out, he says: "In most cases, adults who are less 'swimming' in the internet will also not answer the phone from pollsters. This has been checked in the world, and it was found that this is not necessarily what determines. Ultimately, there is nothing to do, there are populations with under-representation in the polls."

In order to try to reduce the gaps, he notes that from time to time a calibration check can be conducted: "I do completions by phone for the ultra-Orthodox population and the Arab population, to see that I am in the right direction in the samples."

Every method creates its own bias, which always leaves uncertainty. "There are always people who do not answer, who refuse, who are difficult to reach," explains Geva. "We rely on data from the Central Bureau of Statistics (meaning, if 12% of the population are 65+ according to the Bureau, then in the sample this will also be their share). Once this distribution is equal between the sample and the population, we can assume that the sample is representative."

Methodology for every pollster

When it comes to the question of the large gap between the channels, Lazar admits honestly the limits of his knowledge: "When I see that the Likud, for example, moves between 20 and 23 mandates in different polls - for me as a pollster this is the same result. But when talking about 10 to 15 mandates in favor of a certain party - there is no doubt that the poll is no longer based on the same sample."

According to him, the explanation can stem from a combination of factors: a different sampling method, giving different weight to groups in the population, and even the way each institute treats undecideds (a figure that today stands at about 15% of respondents in most polls).

"The Arab society constitutes about 20% of the population," Lazar demonstrates. "You can say - I will give them the full weight in the sample, meaning 20%, and you can say - this is not really 20% in practice on election day, so I will give them a weight of 15% or even 10%. Ultimately, every pollster has their own methodology, and this, in all likelihood, is what creates the difference we see on the screen."

The sample composition and the collection method are only part of the picture, not the full explanation. There may be additional factors for the gap between the groups that were not examined here. For example, differences in the statistical weighting model that each institute operates after data collection, different phrasing of the questions or the order of their presentation, the timing of the poll in relation to political events in that same week, and even the identity of the audience of viewers or surfers of each media outlet, in case part of the sample is recruited through its website or application.