It was September of Fall 2013, and our Social Judgment Research Team in psychology had signed on to be part of the “Many Labs” project, the largest study of reproducibility in the history of psychology. The minute we put up a world map and began sticking in pins representing the collaborating institutions on the project, we were hooked. Ithaca College was one pin. Brazilia University in Brazil was another. HELP University in Malaysia was another. In all, we stuck 36 pins in the map.
Reproducibility is about the extent to which repeating well-defined procedures gives consistent results. It is one of the defining characteristics of science. If a finding can’t be reproduced reliably, it suggests that more research needs to be done to figure out when the effect happens and when it doesn’t. In recent years, there has been increasing attention to the issue of reproducibility across scientific disciplines because results from studies done in one place and time sometimes do not reproduce elsewhere. There also have often been few professional rewards associated with attempting to check the reproducibility of research findings.
The Many Labs project is one of several initiatives of the new Center for Open Science, a nonprofit institution whose purpose is to enhance openness and transparency in science through supporting research on reproducibility. The project examined variation in reproducibility of 13 new and classic effects in psychology across many samples and settings.
The Ithaca College Many Labs group included me and the Social Judgment Research Team, comprised of 10 psychology majors who conduct research for three semesters under my supervision. Our team had signed on to collect data from at least 80 participants, in a laboratory, between the beginning of the fall semester and fall break in 2013. In exchange, we received a copy of the data we collected at the college to analyze and present at the 2014 Annual Convention of the Eastern Psychological Association, a professional conference. Additionally, I co-authored three peer-reviewed research articles about the project’s findings. One article was published April 4 in the Journal of Open Psychology Data, and two other articles will be published soon in another professional research journal, Social Psychology.
The Many Labs project found that 10 of the 13 effects we tested reproduced strongly, and another effect reproduced weakly. Neither the nationality of the participants nor the setting of the laboratory or online studies significantly affected whether the effects reproduced.
A key benefit of the study for the Social Judgment Research Team is that we had the opportunity to analyze and present the data we collected at the college. We conducted the study in a laboratory in Williams Hall, where the participants accessed the online questionnaire used by all collaborating institutions in the project. We reached our target number of the college participants by fall break of 2013. Shortly afterward, we received a copy of the data we collected at the college. We spent several weeks analyzing the college’s data and writing posters about the studies we found most interesting.
Last month, we presented two posters about the Ithaca College Many Labs data at the 2014 Annual Convention of the EPA. We also got to meet one of the heads of the project, Professor Brian Nosek of the University of Virginia, during a presentation he gave about the Center for Open Science. We got to talk with Professor Nosek and the audience about our role in the Many Labs project.
For us, it reinforced how our work at the college on the Many Labs project is part of an important international movement in psychology. Examining the reproducibility of research findings depends on a crowd of researchers willing to do this work, which is not limited to a single project, even one as big as Many Labs. There are thousands of effects in psychology, and much more work remains to be done. In addition to other reproducibility initiatives by the Center for Open Science, there will be a Many Labs 2 and a Many Labs 3 in the fall of 2014. Our research team is already contributing to these projects.
The Many Labs project is headed by Richard Klein and Kate Ratliff of the University of Florida; Michelangelo Vianello of the University of Padua, Italy; and Brian Nosek of the University of Virginia. Ithaca College students who contributed to Many Labs in Fall 2013 are seniors Brittany Carey, Eileen Grandel, Kailyn Mooney, Samantha Towle, Katrina Vega and Chinemenma Udokwu, and juniors Patricia Herrmann, Christine Hill, Angelique Hudson and Shasta Savage.
To learn more about the Many Labs projects and the Center for Open Science, you can go to http://centerforopenscience.org.
