Research First Articles: The Danger of Online Surveys

A Simple Checklist for Evaluating Research

Dimension	Ask Yourself	Details to check
Face Value	What do the results say?	Do the results make sense? How easy are the results to understand?
Context	What do other studies have to say about this issue?	How does this study fit with others? What else does this study have to say? (have results simply been cherry-picked?) Who else can you talk to about this?
Motivation	Who did the research and why?	Who funded the research? Who conducted the research? Is there a vested interest? Are the researchers reporting their findings or attempting to influence an audience?
Quality	How do the researchers know the things they claim?	How many people did they ask? How did they select these people? How did they ask them? What did they ask them? When did they ask them? What analytical techniques were used? How did they interpret the results? Have the data been weighted? Are the researchers clear about the limitations of the research?
Philosophical	Has the research addressed common sources of bias?	How have the researchers avoided common biases such as: Selection bias; Confirmation bias; Publication bias; The Hawthorne Effect; The implicit referent; and The ecological fallacy?

Remember:

Always be wary if you cannot get access to the original research questionnaire, data coding schedule, and the raw data; and
Be wary of research that concludes with the call for more research.

Research First, Level 1 / 320 Manchester Street, PO Box 5596, Papanui, Christchurch 8542
Phone: (508) 4RESEARCH (508) 473 732 info@researchfirst.co.nz