The goal of this assignment is to explore “researcher degrees of freedom”, and how they can lead to false positives. Yes, you will be p-hacking!
Like bad scientists everywhere, you are convinced you your hypothesis is right, and you just need to show that you are right by getting a significant effect. You will do this by trying several tests until you get a significant result.
You will be submitting 2 very short reports; one p-hacked report and one honest report. They have an identical structure. What is different is that the p-hacked report pretends that the data you are presenting and the analyses you conducted are all there is to know. In the real report, you report the full sample, all variables, and what the results of your analysis would be if you did an honest test of the hypothesis with no fancy manipulations.
THE STUDY
You will be working with the data from EXPERIMENT 8 in Bem, D. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100(3), 407-425.
We did a small version of this a few weeks ago. You are asked to visualize a few words, then you are asked to write down as many as you can. For some words (Practice words), you only then (AFTER recall), asked to practice them. For other words (Control words) don’t have the practice sessions.
If you could see into the future, then you would remember more words in the practice trials than the control trials, even though they came AFTER you did the recall.
WRITE UP
P-hacked report
An introduction
Engage in HARKing!! Provide a brief description (one paragraph) of the experimental (alternative) hypothesis. Include a reasonably argued justification for why you think this effect is worth testing. If you include any covariates, or have interactions, come up with a reason for why you predicted them. The Bem paper provides some arguments for why he thinks this will work; you may borrow those.
Methods
Include a brief description of the participants you included, a materials section that includes only the variables you include in your p-hacked analysis final analysis, and the procedures.
Results
You will have a “fake” results section where you report the one result that is the closest to significance as if it was the only test you ran, using APA format.
No discussion necessary
The REAL analysis!
Provide a second report that mirrors the p-hacked one (same sections) but lists the true study in each of the sections, and including all participants and variables, and a simple analysis with no fancy manipulations, just a straight test of the hypothesis.
ADD ONE OR TWO SENTENCES IN AN APPENDIX THAT TELLS US WHAT MANIPULATIONS AND TESTS YOU TRIED SO THAT YOU CAN GET CREDIT FOR YOUR CREATIVITY!
Remember:
Participants are in both conditions so this is a repeated measures (ANOVA) or pair-wise (t-test) test
