psychologists, convenience samples, and really shoddy popular science writing
Psychology has occasionally been called “the study of college undergraduates” and while that would usually be a joke in the psych department, a few writers are raising red flags that it’s too common of a practice and might be affecting the quality of the science. The study they chose to highlight? A survey trying to make the link between someone’s first sexual experience and what sexual activity follows based on 319 heterosexual college students who started having sex only about two years prior to the study and were asked to describe their intimate activities with some very positive and some very negative adjectives from a proscribed list. While the critics ask why the population was so homogeneous and the responses were so limited, this actually makes a lot of sense. If you’re not sure of your hypothesis, you want to have the most uniform samples you can find and limit inherently qualitative feedback into more quantitative form. From there on, you can test if the theory holds for more sexually experienced and diverse populations. So why are science writers harping on a perfectly legitimate, well done hypothesis fishing study?
Probably because it’s recent and it found that the students’ first sexual experience tended to be indicative of how they’d describe their future ones. And when limited to the population studied, it does make sense. Many of them are still relatively wet behind the ears and having finally had a real sexual encounter, they’re wondering what others will be like and comparing it to their first as they get more and more experience because it’s usually one of their few points of reference. At the same time, however, as the first experience fades into memory, new highlights come to take its place and a terrible first time gets forgotten in favor of the last mind-blowing experience and that might go on to color future encounters. We could also wonder about couples who lost their virginity to each other and haven’t had sex with anyone else. So why didn’t the researchers take cases like this into account? Well, they’d be outside the scope of the study, which basically just points out the obvious that yes, there’s a mental link between what you thought of your first time and your future preferences and expectations, as it applies to the sample population.
And that last phrase is really the crux of the matter because while human sexuality is so diverse and complex that questions about it could easily fuel centuries of studies and experiments, the pool of people willing to be studied is limited and the external factors they’d bring into the study makes it complicated to tease out complex and minute differences that might hint at something more, something that merits further research. College undergraduates are easy to recruit, easy to find close to the researchers’ labs, and fairly easy to homogenize, so they make for a simple, convenient set of test subjects in pilot studies. They’re a classic go-to convenience sample, and if you want to study special populations, you’ll go and study those special populations when you have the resources to do so. It’s just not fair to expect a narrow study to account for everything and use it a s springboard to pontificate on the limited utility of convenience sampling in basic psychology published for the public. And here the media has to take some heat as well.
How many pop sci writers just copy and paste the press release? How many of them wrote click bait headlines that sound as if an exhaustive study settled the question of just how special your first time is to you and what role it plays in your sex life? And how many of them trying their best to be contrarians put words in the researchers’ mouths and criticized them for making claims not actually made by the study? My guess? Quite a few. In fact, the links to a critical review of three other studies in the referenced critique were papers uncritically hyped by the media to become the viral stories they became. We can certainly argue about how much psychologists are relying on convenience samples of white, college educated students in the West, and what this does to the field as a whole. However, if the initial studies seem to be suffering from a bad sample or are way too limited to be applied outside of a very narrow socioeconomic group, the media klaxon is making the problem a hundred times worse. For writers to then wag their finger at the scientists, saying “tsk, tsk on your sampling techniques” without acknowledging that their colleagues have been running away with inconclusive and narrow studies for years is very disingenuous.
