This is the second in a series of pieces I have been engaged by The Elephant to write on the place of surveys in the forthcoming election. As I noted in my first piece, I do so as objectively as I can, although for the sake of transparency I again declare that I am currently engaged as a research analyst for one of the firms whose results I review here (Trends and Insights for Africa, or TIFA).
A handful of recent polls
Since my previous piece was published at the beginning of April, the results of three credible surveys were published through the middle of May: those released by the Radio Africa Group in The Star on 18 April, by Trends and Insights for Africa (TIFA) on 5 May, and by Infotrak (sponsored by the Nation Media Group) on 12-13 May. While the first two indicate that Deputy President William Ruto was leading former Prime Minister Raila Odinga in the presidential race by 5 and 7 per cent respectively, they differ somewhat in important details, even beyond their respective headlines. They are also at odds with the most recent of these three surveys: those of Infotrak which have the two in a dead heat at 42 per cent each. I will begin with a visual comparison of the figures from the first two polls.
| Survey Firm | Sample Size/No. of Counties | Data Collection Dates | Ruto | Odinga | Others | Undecided/NR |
| Radio Africa/The Star | 4,057 / 47 | 1-5 April 2022 | 46% | 41% | 1% | 12% |
| TIFA Research | 2,033 / 47 | 22-26 April 2022 | 39% | 32% | 1% | 28% |
| Difference | -7% | -9% | +2% | +14% |
The first point to make is that although the TIFA results gave Ruto a larger lead margin (7 per cent vs. 5 per cent), the difference may not be significant, given that it falls almost entirely within their respective margins of error of around +/-2 per cent. The main difference is a whopping 16 per cent variance in the proportion who failed/declined to answer the presidential choice question. Two obvious questions arising from this are: (1) what might explain this gap, and (2) can any “deeper” analysis of the data suggest the “leanings” of these “silent” respondents?
Regarding the first question, no clear answer is apparent. However, it may be relevant that the Radio Africa poll was conducted by sending text messages to the sample derived from their data-base of phone numbers, whereas TIFA’s was conducted through the more standard Computer-Assisted-Telephonic-Interview (CATI) method (i.e., through live phone calls), based on the (comparable) TIFA data-base. Still, why this would make respondents in the latter survey less willing to reveal their presidential preferences is unclear.
Another puzzling issue has to do with Radio Africa’s stated margin of error, which is given as +/ 4.5 per cent. Yet with a reported sample size of 4,497, based on some 22 million registered voters (a figure against which the data were reportedly weighted), the margin of error is actually only +/-1.5 per cent. Why would the “Star Team” who produced the story want to make their findings look less precise than they are? (I made two phone calls to journalists at The Star about this but neither was able to provide a definitive explanation.)
Further, while the article by The Star made reference to findings from two previous similar Radio Africa surveys – showing Ruto leading Odinga by a huge 29 per cent last July (i.e., 43-14 per cent), and Odinga taking the lead for the first time in March, by 4 per cent (i.e., 47-43 per cent)—no suggestion was offered as to what could account for Ruto’s regaining the lead he previously enjoyed. Indeed, even when reporting the results in March, all The Star could offer was that “Poll ratings can go up and down so Radio Africa will continue to conduct its monthly opinion polls up to the August 9 elections.” In any case, it remains to be seen whether the just concluded party nominations and running mate selections will further reshape the race.
Why would the “Star Team” who produced the story want to make their findings look less precise than they are?
In addition, while the text of the article reported some results by region (i.e., North Rift, South Rift, Central, Upper Eastern, Lower Eastern, Nyanza, Western and Coast), the kind of quite useful table of results that had been included in the mid-March story was absent, as were any explicit comparisons of results at this sub-national level, leaving such calculations up to its readers to make—assuming they had kept the March results for such a purpose: e.g., an increase in support for Ruto in North Rift (from 63 per cent in March to 69 per cent now) but basically no change in Central (57 per cent in both March and April). While such changes may help to explain the overall result—the re-establishment of Ruto’s lead – it would be useful if the respective margins of error for each of the regions was also noted so that such sub-national changes from one poll to another could be put in a clear statistical perspective. But the earlier point remains: could the “Star Team” have provided any explanation as to why Odinga’s rating in Nyanza fell by 11 per cent in a little over one month?
Yet another methodological point is important to make here. If Nyanza constitutes about 13 per cent of the total sample, that generates a margin of error of +/-6 per cent, which means Odinga’s actual proportion in March (shown in The Star’s table as 78 per cent) could have been anywhere in the range of 71-84 per cent, and in April (shown as 67 per cent) in the range of 61-73 per cent. Note here that at the low end of the March figure and the high end of the April figure there is an overlap of 2 per cent, meaning that it is possible—though unlikely—that his “true” Nyanza figures did not change at all over this period.
Could the “Star Team” have provided any explanation as to why Odinga’s rating in Nyanza fell by 11 per cent in a little over one month?
Another point: according to the Market Survey Research Association (MSRA) Guidelines, its members should interview at least 1,500 respondents in any survey reporting national (presidential) voting intention results, and while there is no such minimum for any sub-national polls (county, parliamentary constituency, ward), the margin of error should be made clear. There is, of course, no such requirement for the Radio Africa survey since the regions for which results are reported are not electoral units, even were this news organization an MSRA member (which it is not).
Stop press—another poll is out!
As I was writing this piece, the results of another national survey were released, this one by Infotrak, commissioned by the Nation Media Group. Its headline findings may be compared with TIFA’s in the same way as the latter are compared with Radio Africa’s above:
| Survey Firm | Sample Size/No. of Counties | Data Collection Dates | Ruto | Odinga | Others | Undecided/NR |
| TIFA Research | 2,033 / 47 | 22-26 April 2022 | 39% | 32% | 1% | 28% |
| Infotrak | 2,400 / 47 | 8-9 May 2022 | 42% | 42% | 1% | 15% |
| Difference | +3% | +10% | 0% | -13% |
Before addressing the main contrasts in these results, several points regarding methodology should be made. As Macharia Gaitho, writing an accompanying piece for the Daily Nation, pointed out, “The dates the data was [sic] collected will always have a bearing on the outcome in a fluid political situation, but unless there were very major shifts and realignments in the intervening period, even a fortnight between two polls cannot account for such variations.” This assertion makes sense given the absence of any dramatic events relevant to the fortunes of the two main contenders during the period between the two polls—the departure of Governors Alfred Mutua and Amason Kingi from Azimio to Kenya Kwanza coming just after data collection for the Nation/Infotrak poll. And he went on to add: “Other factors which influence the outcome of an opinion poll are methodology and sampling.” Note that Gaitho had raised some of these same important issues in a Daily Nation piece he authored just before the August 2017 election which I responded to and published in the Nation a few days later, identifying issues that I felt he had not sufficiently addressed.
Leaving aside the semantic error that sampling and methodology are distinct (since sampling is an inherent aspect of methodology), the factors he cites are certainly relevant: sample size and distribution across the country.
Sample size in this case is not an issue, since despite Infotrak’s being slightly larger, the respective margins of error are nearly identical: +/-2 per cent vs. +/-2.17 per cent. Further, even though Infotrak’s question wording differs slightly from TIFA’s (“…who would you vote for…?” vs. “…who do you want Kenya’s next president to be?”), it can be assumed that much of the contrast between the two surveys is a consequence of sampling. Simply put: how each sample was distributed across the country, and what the achieved distribution (whether through the “pot luck” willingness of those selected to participate, or through post-data collection weighting—or some combination of both) is for each demographic variable that might be relevant to the issue at hand: presidential candidate preference.
Unfortunately, and in contrast to TIFA’s release, the Nation gives us no basis for comparing these two samples in demographic terms. That is, while TIFA included figures for gender, age groups, and education level (the latter required by the relevant law enacted in 2012)—as well as the proportion allocated to each of the nine “zones” for which results were reported—we have no such data from the Nation (whatever Infotrak may have provided).
Of course, for most political surveys in Kenya the most salient variable in assessing the representative accuracy of a sample is its ethnic distribution. Most survey firms collect data on the ethnic make-up of its sample but, as far as I know, no survey firm has ever published it nor does the relevant Act require it. Presumably, this is for reasons of “sensitivity”—a form of “self-censorship” that appears universally accepted. This is not to suggest that ethnicity explains “everything” about the achieved results, but it is critical when gauging whether samples are truly representative. For example, in TIFA’s survey, well over half—but nowhere near all—of Ruto’s and Odinga’s “home” ethnic groups (i.e., the Kalenjin and Luo) expressed support for their respective presidential candidates. Given this reality, it is impossible to judge the comparability of the two samples involved here without knowing what proportion of the total sample in each survey is comprised of the country’s main ethnic groups. Specifically, was there a “sufficient” number of Kalenjin in the Infotrak sample, and a “sufficient” number of Luo in TIFA’s (and Radio Africa’s)? Of course, such figures should reflect “correct” random sampling based on the geographical distribution of registered voters according to the IEBC, rather than any “search” for these ethnic proportions.
Of course, for most political surveys in Kenya the most salient variable in assessing the representative accuracy of a sample is its ethnic distribution.
It should also be recognized that whether using the eight pre-2010 Constitution provinces, or TIFA’s nine “zones”, none of them is mono-ethnic, with some of the more homogeneous—Central, Nyanza and Western—having less than 90 per cent of their dominant ethnic groups (Kikuyu, Luo and Luhya, respectively). TIFA’s data indicates Lower Eastern is the most ethnically homogenous “zone” although even here, Kalonzo Musyoka, the leading Kamba candidate, only polled at 15 per cent support.
Another problem arises when using survey results to predict election outcomes. While the presidential contest results of both surveys were generally presented by the media as reflecting actual 9 August ballot choices, in neither case was it reported whether all respondents were registered voters, and among those who claimed to be (assuming they were asked), how “certain” they were that they would actually vote on Election Day. Despite this, the main Nation newspaper article reporting the results refers to the survey’s respondents as “voters”. This raises the possibility that actual voter turnout (that is known to vary across the country in every election) will deviate from the samples of these three surveys, even if they all claim to have used the distribution of registered voters as their sampling “universe”.
Since this cannot be precisely known in advance even much closer to the election, it is misleading to use such survey results to suggest, let alone predict, actual outcomes, as the Daily Nation did in its front-page caption by stating that neither Ruto nor Odinga “has enough backing to cross the 50%+1 threshold to win”. (It may be assumed that those involved in producing these stories are also aware of the additional requirement of obtaining at least 25 per cent in at least 24 of the 47 counties.)
But there is an even more blatant flaw in this “run-off contest” statement: an actual voter who wishes his/her vote to count cannot be “undecided” or “refuse to answer” as one can in a survey interview, since voters must choose from actual ballot choices. In response to one of the several 2013 presidential election petitions, the Supreme Court ruled that spoiled ballots are removed from the count of “total votes cast”, so that the denominator of the calculation for each candidate is based on total valid votes cast, not the total number of people who walked into a polling station.
TIFA’s data indicates Lower Eastern is the most ethnically homogenous “zone” although even here, Kalonzo Musyoka, the leading Kamba candidate, only polled at 15 per cent support.
Based on this reality, for example, the Infotrak results imply that each of the main candidates “would get” 49 per cent, since only about 1,008 of the reported total sample of 2,400 respondents mentioned Odinga and Ruto, and this figure should be divided by roughly 2,040, which is the figure we are left with after subtracting those who said they were undecided or who refused to answer the question—about 360 respondents.
In other words, even if these top two candidates are nearly tied on 9 August, it would seem that a run-off contest would be unnecessary unless the combined figure of all the other presidential candidates exceeds 2 or 3 per cent—a much more likely prospect should Kalonzo Musyoka insist on “going it alone”.
Another contrast between the Infotrak and TIFA polls is important to point out, as it, too, could help to explain their contrasting results.
In terms of the distribution their samples, TIFA uses nine ethno-political “zones” while Infotrak (like Radio Africa) continues to use the former eight provinces. As such, the only sub-national results that can be compared (since they are used by both firms) are: Nairobi, Coast, Western and Nyanza. The table below shows the figures for these four units (comparing the TIFA figure on the left with Infotrak’s on the right, and the difference in parentheses on the far right):
| Nairobi TIFA / Infotrak |
Coast TIFA / Infotrak |
Nyanza TIFA / Infotrak |
Western TIFA / Infotrak |
|
| Ruto | 25% / 33% (+8%) | 26% / 29% (+3%) | 21% / 18% (-3%) | 37% / 33% (-4%) |
| Odinga | 40% / 51% (+11%) | 36% / 55% (+19%) | 56% / 72% (+16%) | 29% / 48% (+19%) |
Even setting aside the higher error margins for each of these regions, which range between about +/-6 and 7 per cent—and thus equal to 12 per cent and 14 per cent spreads, based on their respective sub-national sample sizes—these contrasts are remarkable, especially the larger figures for Odinga in the Infotrak survey. (Note that given the lower Infotrak figures for “undecided” and “no response”—10 per cent and 5 per cent, respectively, as compared to TIFA’s 16 per cent and 12 per cent—Infotrak’s overall figures for both candidates are higher, as is also the case in the comparison of Radio Africa’s figures with TIFA’s.)
While the contrasting figures for Nairobi are minimal, since the 8 per cent difference between Infotrak’s and TIFA’s figure for Ruto is only 3 per cent lower than the difference in Odinga’s numbers for the other three sub-national units (Coast, Nyanza and Western), Odinga’s “Infotrak advantage” is much higher: 16 per cent, 19 per cent, and 23 per cent, respectively. As noted, while TIFA provides the proportions allocated in its sample to each of the nine zones for which it presents results, Infotrak—or at least the Daily Nation—does not. But assuming they were roughly similar—and, as indicated, even taking the higher error margins for each into account—Gaitho’s summary point stands: that such disparate results cannot be accounted for by a minor discrepancy in the dates of the two surveys, given that no major events occurred that might have caused any significant shifts in attitudes towards either of the two main presidential candidates.
In connection with such contrasts between Infotrak and other credible survey firms in Kenya, a little history may be useful. For example, in their last surveys before the contested 2007 election, The Steadman Group gave Odinga a 2 per cent advantage over Kibaki, while Infotrak gave him a 10 per cent lead. Just before the 2013 election, Infotrak “predicted” an outright win for Odinga, whereas Ipsos’ results indicated that neither candidate would achieve this in the first round. A little earlier, an Infotrak survey conducted at the end of December 2012 and into the first few days of 2013 gave Odinga a 12 per cent advantage over Uhuru Kenyatta (51 per cent to 39 per cent), whereas an Ipsos poll conducted only about two weeks later gave the former only a 6 per cent advantage (46 per cent to 40 per cent). Similarly, in their final surveys before the 2017 election late July, Infotrak had Odinga and Kenyatta in a virtual tie (47 per cent vs. 46 per cent) —on which basis it suggested that no one would win on the first round—whereas Ipsos gave Kenyatta a clear outright win: 52 per cent vs. 48 per cent, excluding those who claimed to be undecided or who declined to reveal their preference (which when included, generated a 47 per cent vs. 43 per cent advantage for Kenyatta), although leaving room for some minor deviation from these figures based on differential voter turnout across the country. In fact, Ipsos offered four possible voter turnout scenarios, none of which put Odinga closer than 4 per cent behind Kenyatta.
Of course, given the disputed nature of the official results in all three of these elections, it is impossible to know which survey firm’s results were closer to “the truth”. But they do reveal a clear pattern: that Infotrak has consistently given more positive results to Odinga than any of the other reputable survey firms in the country. I should stress, however, that this “track record” should not be the sole basis for dismissing Infotrak’s current figures, but it does underscore Gaitho’s point that whenever there are differences in survey firms’ results that go beyond the stated margins of error, additional scrutiny is warranted—preferably to a degree that goes beyond what the Nation Media Group offered its readers/viewers on this occasion. Of course, given the fact that they sponsored the survey, such rigorous scrutiny might have been considered “inappropriate” at best.
On the other hand, it should also be recalled that in the polls conducted just before the August, 2010 constitutional referendum, Infotrak produced results that were slightly more accurate than those of Synovate—and there were no claims of any “rigging” in response to the official results.
In any event, the promise of continued polling by at least these three firms—Infotrak, TIFA and Radio Africa—should not only provide Kenyans with an evolving picture of the possible electoral fortunes of particular candidates and political parties/coalitions (as well as identifying the issues motivating voters at both the national and sub-national levels), but also invite them to more thoroughly scrutinize the performance of these firms and of any others that may appear and attain any serious media coverage. This is so even if the announcement of deputy presidential running-mates will make the next set of polls non-comparable with those examined here.
Of course, given the disputed nature of the official results in all three of these elections, it is impossible to know which survey firm’s results were closer to “the truth”.
The most useful data, however, would be the actual results announced by IEBC about which no credibility doubts are raised, so that only differential voter turnout would have to be taken into account in assessing the performance of the pollsters in their final survey rounds.
A concluding comparison: in the recent, second round run-off French presidential election, the final national poll had President Emmanuel Macron defeating Marine Le Pen by 56 per cent to 44 per cent, and the official (uncontested) results gave him a 58.5 per cent to 41.5 per cent victory (just within the poll’s margin of error). Given that, so far at least, there is no evidence that Kenyans lie any more than French people do when answering survey questions, let us hope that Kenyan survey firms can both individually and collectively achieve such accuracy.
Post-script
Just after completing this piece, on May 18 TIFA released the results of another CATI survey it had conducted the day before, comprised of 1,719 respondents. This came in the immediate wake of two days of drama: first, on 15 May, the announcement by Kenya Kwanza Alliance’s presidential candidate, DP William Ruto, of his coalition’s deputy president running mate, Mathira Member of Parliament Rigathi Gachagua, and on the following day, the announcement by Azimio la Umoja’s Raila Odinga of former member of parliament and cabinet minister Martha Karua as his running mate.
The most useful data, however, would be the actual results announced by IEBC about which no credibility doubts are raised.
The two main questions it sought to answer were: (1) How many Kenyans were aware of each of these running mates? (2) What were their presidential voting intentions?
For whatever reasons, it emerged than far more people were aware of Karua’s selection than of Gachagua’s (85 per cent vs. 59 per cent), a pattern which was replicated among those who expressed the intention to vote for Odinga or Ruto, respectively (90 per cent vs. 69 per cent). By contrast, awareness of Kalonzo Musyoka’s choice of running mate, Andrew Sunkuli, was far lower among both the general public and among those (few) who reported an intention to vote for Musyoka (21 per cent and 38 per cent, respectively).
But as expected, it was the results of the second issue that attracted most attention, which gave Odinga a modest but measurable lead over Ruto: 39 per cent to 35 per cent. The central question these figures raised was whether Odinga’s jump into the lead, beyond the tie that Infotrak had reported just one week earlier, was to any degree a consequence of the identification of these two running mates, a question that will be addressed in the next piece in this series, by which time it is hoped at least one additional poll would have been conducted so as to confirm whether TIFA’s most recent figures do indeed represent a major shift in the presidential electoral terrain.
