Rethinking Participant Quality in Behavioral Research

This title was summarized by AI from the post below.

One of the things I admire about behavioral science is our obsession with measurement. Some of us spend careers refining items and scales to measure depression, anxiety, political ideology, intelligence, personality, loneliness, you name it. These latent constructs are necessarily measured using multiple-item scales that get rigorously tested for internal and external validity. None of them is of course expected to be captured by a single item. We infer extraversion or executive function by aggregating multiple imperfect indicators. We know that each carries a little bit of signal and a little bit of noise. It would be ridiculous to think that you can measure, say, someone’s personality by asking them one question… right? Yet, when it comes to the very people we’re studying — specifically, whether they’re good enough to participate in our research — we kind of abandon this logic. Somewhere along the way, we collectively started hoping that a single indicator (e.g., attention check) could tell us whether a participant was “good” or “bad.” I think it’s time we rethink this. If you’re a behavioral researcher conducting online studies — especially beyond the West — I wrote a new piece you might find interesting. I share a practical approach to participant quality that my team recommends as of 2026, after helping a few hundred researchers run behavioral studies across 20+ non-Western countries. In a nutshell, this piece will encourage you to: -> stop relying on any single quality check as a definitive marker of participant quality; -> look at aggregated signal across consistency, duration, content of responses, cognitive engagement - and yes, special checks, too; -> reconsider your loyalty for Instruction Manipulation Checks (if you had one in the first place (I once did)); -> ALWAYS pilot first, and better pilot before preregistering, so you have the right kind of freedom to decide which quality indicators will actually be useful for your analyses; -> have the discipline to validate your decisions empirically: in the final dataset, compare both known and novel effects across the people you’ve marked as “high” and “low” quality— this way, you’ll know if you did it the right way; -> and whenever possible, feed those quality labels back to your recruitment platform. (At Besample, for example, we reward researchers for doing so because every piece of feedback helps improve our quality models.) Find the link in the comments 🔽 Also, if you want to exchange ideas, brainstorm data quality measures for your upcoming study, or even get ready-to-use templates, I run complimentary quality control consultations — they help us learn where our customers stand and help make their studies a success. Link also in comments.

  • No alternative text description for this image

I’m so excited to see this published, Elena!! And so glad you called out the importance of piloting before preregistration. Can't wait to share what we've been working on to help researchers get more out of their pilot studies. (Also, the meme is perfect!)

See more comments

To view or add a comment, sign in

Explore content categories