When Crowds Judge Science: Citizens’ Evaluations of Project Social Impact

Strategy & Business Policy

Speaker: Henry SAUERMANN 

Professor ESMT Berlin

Videoconference

Abstract

There have long been concerns that the professional scientific system puts too little weight on the social benefits (and costs) of research when setting research agendas. A growing number of mechanisms seek to address this concern, including explicit calls for “broader impacts” by funding agencies as well as participation of non-scientific actors such as patient organizations in policy and funding decisions. The most novel mechanisms allow individual citizens to directly influence research agendas by voting on research proposals or funding research projects with their own money. Although there is a hope that such mechanisms can help “democratize” science and steer research towards greater societal relevance, it is not clear how crowds evaluate the potential social impact of research projects, and what role social impact plays relative to other criteria.

To examine this question, we asked over 2,000 crowd members to evaluate funding proposals written for the general public in the social, natural, and medical sciences. We elicited assessments of different project attributes (e.g., scientific merit, researcher capabilities, social impact), recommendations for funding, as well as real donations using personal money.

Using these data, we first confirm the common assumption that crowd members put a high weight on social impact when evaluating research projects. We then analyze how judgments of social impact relate to individual level attributes such as level of education, income, and personal relevance of the project topic. Qualitative data provide additional insights into what exactly people think about when judging projects’ social impact. Finally, although social impact is inherently subjective, we seek to gain initial insights into the accuracy of crowd judgments. Towards this end, we compare crowd judgments to an “objective” indicator of problem importance available in the medical sciences (the Burden of Disease, which reflects both the number of people affected by a disease and the severity of the resulting conditions).

Our study contributes to the literatures in science and innovation management as well as crowdsourcing. We also discuss implications for scientists, funding agencies, and policy makers.