//
you're reading...
Musings

Reporting Data In Singapore? Beware The Sample

Reporting Data In Singapore? Beware The Sample

Follow my Facebook page to keep track of updates, or subscribe to my newsletter.

There has been, in the past weeks, a number of news reports in the Singaporean media covering different issues, but which follow a familiar pattern: A survey or data analysis is conducted, supposedly representative conclusions are conveniently drawn, and the headline then leads with a top-line, sensational statistic. “Nearly one in three graduates quit first job in less than a year” (Sept. 22), according to TODAY and job site Monster.com. “76 per cent of Singaporeans positive towards new F1 deal” (Sept. 20), according to TODAY and global media monitoring house Meltwater. And “Marketing Magazine“, in an article about online sentiments of the recently-concluded presidential elections, cited Meltwater data that 83 per cent of these sentiments were “negative” (Sept. 12).

Yet little attention has been paid to the samples of the surveys and data analyses, which should immediately cast doubt over the validity of the aforementioned conclusions and headlines. Statistically, because it is rarely feasible or cost-effective to reach out to a large population, sampling of a subset – randomly and properly conducted to minimise bias – therefore allows for observations to be made about the population. Take for instance, a government which wants to regularly ascertain the extent to which its constituents approve of its performance. Since it cannot hold elections for the population all the time, the government could instead survey a small but representative sample of (likely) voters, to keep track of its approval ratings.

In scenarios like this, sample selection is of paramount importance. Choose too many voters who are affiliated with the political party in power, and approval could be overstated. Choose too many who are from a particular demographic or socio-economic group, and the results could be skewed. And this is exactly the mistake we see, of the recent news reports reporting data:

– “Nearly one in three graduates quit first job in less than a year” (TODAY): The sample of 2,368 respondents – 500 of whom were Singaporeans – are jobseekers (and employers) on a job site. Two issues with the data reporting. First, on sampling, “nearly one in three graduates” is simply not accurate, without the inclusion of those who are actually employed. Second, on utility, what and where is the context? Even if the one-in-three figure is accurate, how does it compare to Singaporeans of other age-groups? Over time, or across industries? How does one use the data and its conclusions?

– “76 per cent of Singaporeans positive towards new F1 deal” (TODAY): Meltwater “tracked online reactions in the lead-up to and during the race weekend”. Notwithstanding the lack of information about methodology, in terms of how it determined if the reactions were “positive”, “negative”, or “neutral”, the sample of Internet users who follow and post news or updates related to Formula 1 are already more likely to have an interest in the sport, and hence be more “positive”. Furthermore, the firm is unlikely to have access to private or protected social media accounts, which inevitably skews the findings.

– “Online sentiments surrounding Halimah Yacob’s presidential walkover” (“Marketing Magazine“): The data was provided by Meltwater, and in this example the issue of private or protected social media accounts is more pertinent. Even more so, if closed-group social media – such as WhatsApp or Telegram – are preferred for news and opinions.

What is more frustrating is the templatised manner in which the conclusions and headlines are substantiated. The news reports either pull anecdotes and social media excerpts (as it was for the Formula 1 and presidential election articles), or seek quotes from the firms which produced the data, as well as token quotes from “the public” to fit the narrative (as it was for the graduate survey and the Formula 1 article). In line with the findings that many graduates reportedly quit in less than a year, with “the lack of professional growth as a main reason for quitting”, the two interviewees in the news report neatly fit this pre-determined template.

Perhaps the assumption is that readers should be more cognisant, or be more sceptical of such data reporting. And while we as readers should be responsible too, the responsibility to responsibly present or report such data – I believe – primarily and inter-relatedly falls on the firms producing the data, together with the media. Firms are incentivised to raise their profile and to increase the traction of their surveys or data analyses, yet they should be honest about the limitations of their endeavours. The media, in this vein, should go beyond the mere regurgitation of top-line statistics or press releases to provide useful context, and should not hesitate to hold the firms to account.

About guanyinmiao

A man of knowledge lives by acting, not by thinking about acting. Carlos Castaneda.

Discussion

3 thoughts on “Reporting Data In Singapore? Beware The Sample

  1. Data sampling is a big issue nowadays. Different reports claim different things based on a small amount of samples taken which cannot be considered to be 100% true. We need to be cautious.

    Posted by Achal | September 27, 2017, 3:19 pm

Trackbacks/Pingbacks

  1. Pingback: “Lies, Damned Lies, And Statistics”: Surveys On Private School Graduates And Foreign Domestic Workers | guanyinmiao's musings - December 11, 2017

  2. Pingback: Success Of New National Conversation Depends On Participation Of “Naysayers” And The Less Privileged | guanyinmiao's musings - May 19, 2018

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Follow guanyinmiao's musings on WordPress.com

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 1,363 other followers

Twitter

%d bloggers like this: