A university module on philanthropy, impact investing, and social enterprises gave me my first insights on performance measurement and management or PMM (and the module also led to my final-year thesis, an exploratory study on how non-profits organisations (NGOs) in Singapore use PMM), highlighting the importance of evidence-based reporting in non-profits. That public service organisations – not just NGOs – should evaluate how effective its programmes or services have been may seem like a straightforward proposition, yet such evaluations are rarely conducted or publicised in Singapore, as it is in the United States.
Since his inauguration, President Barack Obama has highlighted the role of social science evidence, and 700 social intervention programmes have therefore emerged through six evidence-based initiatives: infant and child health and development, pre-school education, K-12 education, pregnancy and sexually transmitted diseases among teens, student performance, as well as community college education and job training. And in their exposition of these initiatives in “Show Me The Evidence: Obama’s Fight For Rigour and Results In Social Policy”, Ron Haskins and Greg Margolis argue that unless the models and results of public policies are tested and studied, the effectiveness of a programme or service cannot be fairly ascertained.
Take for instance an intervention programme for at-risk youths in Singapore. An NGO provides after-school and counselling services over a one-year period, and claims that the programme has been effective because the participating youths – through a satisfaction survey – said they enjoyed the activities, and that the test-scores of these youths have improved too. Some scepticism, nonetheless, is in order, because of a flawed evaluation method (likely self-response bias, wherein respondents might have been prompted to give more positive answers) and the absence of a counterfactual (it is not clear whether the organisation can claim full credit for the impact).
In this vein randomised controlled trials, or RCTs, as well as other quasi-experimental methods have been mooted as more rigorous strategies. Through an RCT, where participants are randomly allocated into “treatment” and “control” (or “no treatment”) groups – with statistical analyses to check that these two groups, on average, have the same characteristics before treatment – a non-profit would be able to quantify the effect of its programmes and services. Still, Haskins and Margolis highlighted challenges in the book:
– The “ethical argument” that valuable treatment is withheld from the “control group”;
– That policy implementation based on RCTs is not as straightforward, and the tangential point that “evaluations must provide continuous information to programme officials so that they can make adjustments in their programme in response to an evaluation’s findings”, signalling the need for capacity-building undertakings too; and
– That NPOs may be too busy “trying to run their programme well that they have neither the time nor the resources to support expensive and time-consuming random assignment evaluations”.
If PMM and evidence-based policymaking were to take root in Singapore, these challenges will also surface here, and it is not clear whether we have the necessary remedies. And moreover, no panacea appears to exist for these challenges.
The book is a dense read. A substantial portion is dedicated to legislative difficulties in the United States, which are not necessarily applicable to the Singapore context. But the techniques used – such as tiered competitive grants (giving higher amounts of grants to NPOs which provide more evidence of their programmes), specifying that a percentage of disbursed funds must be used for research, evaluation, and capacity building, and the transparent use of review panels in the selection process – can be considered in our public policy sphere.
As it is in the United States, many in Singapore “have become accustomed to thinking, on the basis of anecdotal and other unreliable evidence, that their programmes are working well”. Notwithstanding the difficulty of gathering data and information in Singapore, developing a culture of evidence-based policymaking can – in the long run – enrich policy discourse.