There has been, in the past weeks, a number of news reports in the Singaporean media covering different issues, but which follow a familiar pattern: A survey or data analysis is conducted, supposedly representative conclusions are conveniently drawn, and the headline then leads with a top-line, sensational statistic. “Nearly one in three graduates quit first job in less than a year” (Sept. 22), according to TODAY and job site Monster.com. “76 per cent of Singaporeans positive towards new F1 deal” (Sept. 20), according to TODAY and global media monitoring house Meltwater. And “Marketing Magazine“, in an article about online sentiments of the recently-concluded presidential elections, cited Meltwater data that 83 per cent of these sentiments were “negative” (Sept. 12).
Yet little attention has been paid to the samples of the surveys and data analyses, which should immediately cast doubt over the validity of the aforementioned conclusions and headlines. Statistically, because it is rarely feasible or cost-effective to reach out to a large population, sampling of a subset – randomly and properly conducted to minimise bias – therefore allows for observations to be made about the population. Take for instance, a government which wants to regularly ascertain the extent to which its constituents approve of its performance. Since it cannot hold elections for the population all the time, the government could instead survey a small but representative sample of (likely) voters, to keep track of its approval ratings.
In scenarios like this, sample selection is of paramount importance. Choose too many voters who are affiliated with the political party in power, and approval could be overstated. Choose too many who are from a particular demographic or socio-economic group, and the results could be skewed. And this is exactly the mistake we see, of the recent news reports reporting data:
– “Nearly one in three graduates quit first job in less than a year” (TODAY): The sample of 2,368 respondents – 500 of whom were Singaporeans – are jobseekers (and employers) on a job site. Two issues with the data reporting. First, on sampling, “nearly one in three graduates” is simply not accurate, without the inclusion of those who are actually employed. Second, on utility, what and where is the context? Even if the one-in-three figure is accurate, how does it compare to Singaporeans of other age-groups? Over time, or across industries? How does one use the data and its conclusions?
– “76 per cent of Singaporeans positive towards new F1 deal” (TODAY): Meltwater “tracked online reactions in the lead-up to and during the race weekend”. Notwithstanding the lack of information about methodology, in terms of how it determined if the reactions were “positive”, “negative”, or “neutral”, the sample of Internet users who follow and post news or updates related to Formula 1 are already more likely to have an interest in the sport, and hence be more “positive”. Furthermore, the firm is unlikely to have access to private or protected social media accounts, which inevitably skews the findings.
– “Online sentiments surrounding Halimah Yacob’s presidential walkover” (“Marketing Magazine“): The data was provided by Meltwater, and in this example the issue of private or protected social media accounts is more pertinent. Even more so, if closed-group social media – such as WhatsApp or Telegram – are preferred for news and opinions.
What is more frustrating is the templatised manner in which the conclusions and headlines are substantiated. The news reports either pull anecdotes and social media excerpts (as it was for the Formula 1 and presidential election articles), or seek quotes from the firms which produced the data, as well as token quotes from “the public” to fit the narrative (as it was for the graduate survey and the Formula 1 article). In line with the findings that many graduates reportedly quit in less than a year, with “the lack of professional growth as a main reason for quitting”, the two interviewees in the news report neatly fit this pre-determined template.
Perhaps the assumption is that readers should be more cognisant, or be more sceptical of such data reporting. And while we as readers should be responsible too, the responsibility to responsibly present or report such data – I believe – primarily and inter-relatedly falls on the firms producing the data, together with the media. Firms are incentivised to raise their profile and to increase the traction of their surveys or data analyses, yet they should be honest about the limitations of their endeavours. The media, in this vein, should go beyond the mere regurgitation of top-line statistics or press releases to provide useful context, and should not hesitate to hold the firms to account.