Another quick look at Bulgaria’s “Gallup International” (aka Part II)

Another quick look at Bulgaria’s “Gallup International” (aka Part II)

I’ll forgive you if you don’t remember Bulgaria’s Gallup International (no, not that Gallup – the Gallup International “being sued by Gallup Inc. for using its name without authorization”), who very briefly popped into western headlines in March thanks to a Wall Street Journal article.

Published only days before the election, the article alleged that the pro-Kremlin Bulgarian Socialist Party (BSP) received a secret strategy document proposing, among other things, that they “promote exaggerated polling data” to help build momentum for the party and get them elected. Gallup International was specifically named in the article, and referred to by Bulgaria’s former ambassador to Moscow Ilian Vassilev as one of Moscow’s “Bulgarian proxies”; Vassilev, talking about a poll Gallup International published on an apparent lack of NATO support in Bulgaria, accused the polling agency of making a “wrapped-in-secrecy poll [that] had no details on methodology nor funding sources.”

Of course, the WSJ article wasn’t without its critics, myself included. One journalist in Bulgaria said the WSJ article “completely ignored all the internal political factors that led to the landslide victory of [President] Rumen Radev and focused only on Russian influence in elections,” which ultimately, in his opinion “backfired, instead of there being alarms for (legitimate) concerns about Russian involvement in Bulgaria.” (To say nothing of the role of the “crackpot outfit” at RISI apparently being involved in producing the document.)

Debates about the scope and scale of Russian influence in Bulgaria aside, I decided to take a look at polling data in Bulgaria leading up to the parliamentary election on March 26. Is there anything to support this idea that Gallup International and potentially other Bulgarian pollsters have used and promoted “exaggerated polling data” to benefit the BSP?

There might be.

Here’s a table of all the polls I’ve been able to find in 2017 leading up to the election, with levels of party support noted and the final actual election results at the bottom.

bulgaria poll jpg

(larger pdf of same chart here: Bulgaria polls)

Almost all polls from polling firms like Trend Research (3 of 3), Alpha Research (3 of 3, and, full disclosure, whose managing director I interviewed as part of my April piece for Coda Story) and Estat (2 of 3) showed Boyko Borisov’s GERB having a lead.

To break it further down (remembering that GERB ended up winning by 5.6%)

  • Trend Research: average GERB lead of 1.93%
  • Alpha Research: average GERB lead of 2.77%
  • Estat: average GERB lead of 3.47%

But the story’s pretty different with Gallup International, who one Bulgarian source described to me as “the main political/sociological advisor of the [BSP] at election time.” They published four polls during the campaign, only one of which showed GERB in the lead – their final poll, about a week before the vote, which showed GERB with only a 0.6% lead. On average, Gallup International’s polls showed an average BSP lead of 0.7% throughout the campaign.

Again, GERB ended up winning by 5.6%.

The numbers are even stranger for AFIS, a pollster run by Yuriy Aslanov, a sociologist who has sat on the BSP board. Both its polls showed the BSP in the lead by an average of 1.3%.

Again, GERB ended up winning by 5.6%.

In sum – a pollster that’s been accused as being part of an effort to “promote exaggerated polling data” on behalf of the party it’s linked to consistently showed results that played up the support for said party, contradicting most other polling firms’ results throughout the campaign. As a western observer and polling nerd who’s worked for multiple firms in Canada and the UK, I feel I’m in a position to confidently say this isn’t normal.

Of course, this all needs a few caveats. There’s absolutely no evidence of anyone cooking up numbers or anything like that – I am accusing no one of that. I’m also well aware of how statistics work and am well aware of the (however unlikely) possibility that these figures from Gallup International and AFIS are all down to random survey errors or even differences in methodology. Still, something’s off here.

Above all though, I’d stress this message to other journalists who haven’t had the pleasure of drowning a considerable part of their adult lives in SPSS – if some polling numbers look consistently very different from what other polling numbers are saying, ask why. Sometimes it’s an easy answer – a different, new methodology that may or may not pan out, a rogue poll (i.e., the one out of 20) or the polling firm just really sucks. But sometimes it’s not that simple.

Advertisements

A quick look at Bulgaria’s “Gallup International”

A quick look at Bulgaria’s “Gallup International”

Something in this WSJ story about Russian meddling in Bulgaria caught my eye – the bits about “exaggerated polling data” and this rather curiously-named polling firm:

“A Bulgarian polling company, Gallup International, which isn’t related to U.S. pollster Gallup Inc., accurately predicted Mr. Radev’s victory. The company, which is being sued by Gallup Inc. for using its name without authorization, also co-published a February poll that said citizens of four NATO members, including Bulgaria, would choose Russia, rather than NATO, to defend them if they were attacked. Those results were at odds with a similar poll by Gallup Inc., published a few days earlier, showing that most NATO members in Eastern Europe, including Bulgaria, see the alliance as protection.

Gallup International didn’t respond to requests for comment.

“This wrapped-in-secrecy poll had no details on methodology nor funding sources,” said Ilian Vassilev, Bulgaria’s former ambassador to Moscow. “Russian media strategists and their Bulgarian proxies used the Western name to fool people about its credibility and spread their message.””

So I decided to take a quick look at Gallup International’s website, here. Without digging into the actual polls they do themselves, not much to comment on here.

Things are a bit more interesting on the affiliations page.

esomar 2
Screenshot, March 24, 2017

First on their list of affiliations is the WIN/Gallup International, “made up of the 80 largest independent market research and polling firms in their respective countries.” It’s an affiliation I’ve never come across and has several members I’ve either never heard of or are minor players in their respective countries (not including Leger in Canada, who’s well established in Canada and, coincidentally, also part of ESOMAR), which I guess underscores the “independent” part of the description.

But it’s the ESOMAR bit which interests me most. ESOMAR is a large, well-respected international network of market research firms who abide by a code and guidelines on market research. Not everyone’s a member of ESOMAR (in fact, none of the big firms I worked for were members) but it’s a reassuring thing to see on a polling firm’s site if you know nothing about them.

But this is what you get when you do a search for Gallup International in the ESOMAR directory.

esomar 1
Hmm. (Screenshot, March 24, 2017)

I double-checked the list of Bulgarian members to make sure I didn’t miss them. There are eight Bulgarian members of ESOMAR, according to the directory. Gallup International isn’t one of them.

Hmm.