TY - JOUR
AU1 - Klein, Richard A.
AU2 - Vianello, Michelangelo
AU3 - Hasselman, Fred
AU4 - Adams, Byron G.
AU5 - Adams, Reginald B.
AU6 - Alper, Sinan
AU7 - Aveyard, Mark
AU8 - Axt, Jordan R.
AU9 - Babalola, Mayowa T.
AU1 - Bahník, Štěpán
AU1 - Batra, Rishtee
AU1 - Berkics, Mihály
AU1 - Bernstein, Michael J.
AU1 - Berry, Daniel R.
AU1 - Bialobrzeska, Olga
AU1 - Binan, Evans Dami
AU1 - Bocian, Konrad
AU1 - Brandt, Mark J.
AU1 - Busching, Robert
AU2 - Rédei, Anna Cabak
AU2 - Cai, Huajian
AU2 - Cambier, Fanny
AU2 - Cantarero, Katarzyna
AU2 - Carmichael, Cheryl L.
AU2 - Ceric, Francisco
AU2 - Chandler, Jesse
AU2 - Chang, Jen-Ho
AU2 - Chatard, Armand
AU2 - Chen, Eva E.
AU3 - Cheong, Winnee
AU3 - Cicero, David C.
AU3 - Coen, Sharon
AU3 - Coleman, Jennifer A.
AU3 - Collisson, Brian
AU3 - Conway, Morgan A.
AU3 - Corker, Katherine S.
AU3 - Curran, Paul G.
AU3 - Cushman, Fiery
AU3 - Dagona, Zubairu K.
AU4 - Dalgar, Ilker
AU4 - Dalla Rosa, Anna
AU4 - Davis, William E.
AU4 - de Bruijn, Maaike
AU4 - De Schutter, Leander
AU4 - Devos, Thierry
AU4 - de Vries, Marieke
AU4 - Doğulu, Canay
AU4 - Dozo, Nerisa
AU4 - Dukes, Kristin Nicole
AU5 - Dunham, Yarrow
AU5 - Durrheim, Kevin
AU5 - Ebersole, Charles R.
AU5 - Edlund, John E.
AU5 - Eller, Anja
AU5 - English, Alexander Scott
AU5 - Finck, Carolyn
AU5 - Frankowska, Natalia
AU5 - Freyre, Miguel-Ángel
AU5 - Friedman, Mike
AU6 - Galliani, Elisa Maria
AU6 - Gandi, Joshua C.
AU6 - Ghoshal, Tanuka
AU6 - Giessner, Steffen R.
AU6 - Gill, Tripat
AU6 - Gnambs, Timo
AU6 - Gómez, Ángel
AU6 - González, Roberto
AU6 - Graham, Jesse
AU6 - Grahe, Jon E.
AU7 - Grahek, Ivan
AU7 - Green, Eva G. T.
AU7 - Hai, Kakul
AU7 - Haigh, Matthew
AU7 - Haines, Elizabeth L.
AU7 - Hall, Michael P.
AU7 - Heffernan, Marie E.
AU7 - Hicks, Joshua A.
AU7 - Houdek, Petr
AU7 - Huntsinger, Jeffrey R.
AU8 - Huynh, Ho Phi
AU8 - IJzerman, Hans
AU8 - Inbar, Yoel
AU8 - Innes-Ker, Åse H.
AU8 - Jiménez-Leal, William
AU8 - John, Melissa-Sue
AU8 - Joy-Gaba, Jennifer A.
AU8 - Kamiloğlu, Roza G.
AU8 - Kappes, Heather Barry
AU8 - Karabati, Serdar
AU9 - Karick, Haruna
AU9 - Keller, Victor N.
AU9 - Kende, Anna
AU9 - Kervyn, Nicolas
AU9 - Knežević, Goran
AU9 - Kovacs, Carrie
AU9 - Krueger, Lacy E.
AU9 - Kurapov, German
AU9 - Kurtz, Jamie
AU9 - Lakens, Daniël
AU1 - Lazarević, Ljiljana B.
AU1 - Levitan, Carmel A.
AU1 - Lewis, Neil A.
AU1 - Lins, Samuel
AU1 - Lipsey, Nikolette P.
AU1 - Losee, Joy E.
AU1 - Maassen, Esther
AU1 - Maitner, Angela T.
AU1 - Malingumu, Winfrida
AU1 - Mallett, Robyn K.
AU1 - Marotta, Satia A.
AU1 - Međedović, Janko
AU1 - Mena-Pacheco, Fernando
AU1 - Milfont, Taciano L.
AU1 - Morris, Wendy L.
AU1 - Murphy, Sean C.
AU1 - Myachykov, Andriy
AU1 - Neave, Nick
AU1 - Neijenhuijs, Koen
AU1 - Nelson, Anthony J.
AU1 - Neto, Félix
AU1 - Lee Nichols, Austin
AU1 - Ocampo, Aaron
AU1 - O’Donnell, Susan L.
AU1 - Oikawa, Haruka
AU1 - Oikawa, Masanori
AU1 - Ong, Elsie
AU1 - Orosz, Gábor
AU1 - Osowiecka, Malgorzata
AU1 - Packard, Grant
AU1 - Pérez-Sánchez, Rolando
AU1 - Petrović, Boban
AU1 - Pilati, Ronaldo
AU1 - Pinter, Brad
AU1 - Podesta, Lysandra
AU1 - Pogge, Gabrielle
AU1 - Pollmann, Monique M. H.
AU1 - Rutchick, Abraham M.
AU1 - Saavedra, Patricio
AU1 - Saeri, Alexander K.
AU1 - Salomon, Erika
AU1 - Schmidt, Kathleen
AU1 - Schönbrodt, Felix D.
AU1 - Sekerdej, Maciej B.
AU1 - Sirlopú, David
AU1 - Skorinko, Jeanine L. M.
AU1 - Smith, Michael A.
AU1 - Smith-Castro, Vanessa
AU1 - Smolders, Karin C. H. J.
AU1 - Sobkow, Agata
AU1 - Sowden, Walter
AU1 - Spachtholz, Philipp
AU1 - Srivastava, Manini
AU1 - Steiner, Troy G.
AU1 - Stouten, Jeroen
AU1 - Street, Chris N. H.
AU1 - Sundfelt, Oskar K.
AU1 - Szeto, Stephanie
AU1 - Szumowska, Ewa
AU1 - Tang, Andrew C. W.
AU1 - Tanzer, Norbert
AU1 - Tear, Morgan J.
AU1 - Theriault, Jordan
AU1 - Thomae, Manuela
AU1 - Torres, David
AU1 - Traczyk, Jakub
AU1 - Tybur, Joshua M.
AU1 - Ujhelyi, Adrienn
AU1 - van Aert, Robbie C. M.
AU1 - van Assen, Marcel A. L. M.
AU1 - van der Hulst, Marije
AU1 - van Lange, Paul A. M.
AU1 - van ’t Veer, Anna Elisabeth
AU1 - Vásquez- Echeverría, Alejandro
AU1 - Ann Vaughn, Leigh
AU1 - Vázquez, Alexandra
AU1 - Vega, Luis Diego
AU1 - Verniers, Catherine
AU1 - Verschoor, Mark
AU1 - Voermans, Ingrid P. J.
AU1 - Vranka, Marek A.
AU1 - Welch, Cheryl
AU1 - Wichman, Aaron L.
AU1 - Williams, Lisa A.
AU1 - Wood, Michael
AU1 - Woodzicka, Julie A.
AU1 - Wronska, Marta K.
AU1 - Young, Liane
AU1 - Zelenski, John M.
AU1 - Zhijia, Zeng
AU1 - Nosek, Brian A.
AB - We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance (p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion (p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely high-powered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied.
TI - Many Labs 2: Investigating Variation in Replicability Across Samples and Settings
JF - Advances in Methods and Practices in Psychological Science
DO - 10.1177/2515245918810225
DA - 2018-12-01
UR - https://www.deepdyve.com/lp/sage/many-labs-2-investigating-variation-in-replicability-across-samples-RiJ1XbX8M1
SP - 443
EP - 490
VL - 1
IS - 4
DP - DeepDyve
ER -