Should experiments make sense?

Abstract

No[1].

 

Should psychological experiments make sense to the participant? To me, the answer always seemed obvious: of course they need to make sense; from decision making (Daw et al., 2006) to intentional binding (Haggard et al., 2002), from cooperation (Rand et al., 2012) to the Eriksen flanker task (Eriksen & Eriksen, 1974) and many other studies: they all make sense to the participant. In this essay though, I will present a few papers that show that psychological experiments do not need to make sense at all to the participant.

Picture the following situation: you’re in an experimental booth in front of the computer and you see a rapid series of images flash on the screen. At a certain moment in that sequence you know there will be either a chair, a piano, or some visual noise. The images stop and you’re now supposed to say whether you saw a piano, chair, or visual noise at that specific moment. The images are shown rapidly and often you’re sure you didn’t even see anything. And yet, you still have to respond. It makes little sense, you didn’t see anything, you’re purely guessing, and yet (although the participant does not know this), once the data are analysed, participants were actually able to be correct statistically above chance level for the chair and the visual noise. The experiment didn’t make much sense to the participant, because they had to say what they believed they saw, even though they didn’t see anything in one condition (the stimuli were visible for 16.7ms). But of course, from an experimental perspective, this makes sense; in fact, it is expected that the participants are confused and guessing, because that’s what the scientists tried to study (Bode et al., 2012)[2].

Consider another experiment, this time by Moore & Pope (2014). In this study, the participants have to judge whether or not a button press in a video clip was intentional. In the video there is a finger pressing a key of a computer keyboard; a chord is attached to the finger, which could have pulled the finger down (to cause the action). The task is simple: did the person press the button, or was the person’s finger pulled down? Unbeknownst to the participants though, they always saw the same absolutely ambiguous video; there’s just no way of saying either way with any confidence whether the person intentionally presses the button or not.

So again, the experiment makes little sense to the participant. They have to judge something (whether an action was intentional or not) without any relevant cues. Intention depends on many crucial parts, such as the motivation for doing an action, or a person’s personal history in similar situations. So how are you supposed to judge intentions if you have no relevant cues but only see the same video repeated? Shouldn’t the rate just be 50/50? The interesting thing is, no, there is a consistent bias in humans to judge something as being intentional in such ambiguous situations. And so, the fact that this experiment showed a bias towards judging an ambiguous situation as intentional shows that, although the experiment makes little sense to the participant, it makes sense as a study[3]. Finally, there is one experiment that might not make sense to the participant, or even as an experiment per se. This experiment is the Libet experiment.

The Libet experiment also doesn’t necessarily make sense from the participant’s perspective. You're supposed to press a button whenever you feel the urge to move, but there is no reason to do so, there are no incentives. Then, the participants have to judge when they felt this urge according to a clock that is in front of them. The entire experimental procedure is rather odd. Does the ‘urge’ that participants are supposed to report really exist in such a meaningless repetitive task? I still have my doubts. The problem is this: in the Libet experiment the dependent measure is the moment of urge relative to movement onset. The is problematic because there is no way to objectively assess whether or not participants correctly reported the timing of their urge. So we have no idea whether the results from the Libet experiment actually mean anything.

The contrast here to the other two experiments is striking: although the first two might not make any sense to the participants while they are doing the experiment, there is always an objective way to assess one’s DV: in the Bode et al. study we can compare how often participants correctly identified the image in the low visibility condition to chance performance (it’s significantly better); in the Moore & Pope study we can compare the average percentage of reported intentions to the baseline of guessing (it’s significantly higher, suggesting a general bias to assume intentionality in ambiguous situations). But in the Libet study? If the experiment makes no sense to the participants and we don’t have any (even theoretical) possibility of testing whether or not their claims are true (or deviate from some expected figure), then what exactly are we doing here?

And so I think this can answer the initial question: a psychological experiment does not need to make any sense to the participant, as long as there is an objective way to assess the meaningfulness of the outcome measure. Bode et al., and Moore & Pope do this, while Libet et al do not. It’s not that this means that the Libet experiment is wrong, it’s just that we have no way of really knowing. This is of course a general problem with the study of internal sensations, but that doesn’t make the experiment any more sensible.

 

References

Berry, M. V., Brunner, N., Popescu, S., & Shukla, P. (2011). Can apparent superluminal neutrino speeds be explained as a quantum weak measurement. J. Phys. A: Math. Theor, 44, 492001.

Bode, S., Bogler, C., Soon, C. S., & Haynes, J. D. (2012). The neural encoding of guesses in the human brain. Neuroimage, 59(2), 1924-1931.

Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006). Cortical substrates for exploratory decisions in humans. Nature441(7095), 876-879.

Eriksen, B. A., & Eriksen, C. W. (1974). Effects of noise letters upon the identification of a target letter in a nonsearch task. Perception & psychophysics16(1), 143-149.

Haggard, P., Clark, S., & Kalogeras, J. (2002). Voluntary action and conscious awareness. Nature neuroscience5(4), 382-385.

Libet, B., Gleason, C. A., Wright, E. W., & Pearl, D. K. (1983). Time of conscious intention to act in relation to onset of cerebral activity (readiness-potential). Brain, 106(3), 623-642.

Moore, J. W., & Pope, A. (2014). The intentionality bias and schizotypy. The Quarterly Journal of Experimental Psychology, 67(11), 2218-2224.

Rand, D. G., Greene, J. D., & Nowak, M. A. (2012). Spontaneous giving and calculated greed. Nature, 489(7416), 427-430.

 

[1] * Inspired by Berry et al. (2011): https://arxiv.org/pdf/1110.2832v2.pdf

[2] I could equally well have chosen some example from the priming literature, in which certain stimuli are also flashed briefly. The participants are unaware of the stimuli, but these flashed stimuli can nevertheless still influence later decision making

[3] I’m not saying this is a perfect and there are problems (nothing is being manipulated, it’s all correlational; there are no tests to control that this effect is an actual effect and not just an artefact e.g., (if I remember correctly) most people tend to say yes rather than no to questionnaires and the authors didn’t reverse the question). But the experiment is still worthwhile, because it offers a nonverbal paradigm for the study of the intentionality bias, which had previously been established in tasks that relied on verbal descriptions of different scenarios

It's all a sham

An intentional introduction to the intentionality bias