On Being a Sample Size of One

I recently finished reading a statistics “textbook” for fun (“textbook” is in quotes because it was one of those “make learning enjoyable” books, thus diminishing the nerd factor of the achievement). I had always wanted to take a stat course before. Expectation, variance, binomial and Poisson distributions ... I thought it was all pretty cool stuff.

One thing you learn when studying statistics: How to quantify the probability that an observed outcome for a trial/study/experiment is significant. In other words, is it likely that some causal agent A led to some outcome B, or were the study’s results probably due to chance?

One thing that helps in making this determination: sample size. The larger, the better.

Which brings us to my personal knee experiment. What does it mean to be a sample size of one?

Well, from a professional statistician’s viewpoint: not much. A sample size of one is a joke. Where are your control subjects? How could you possibly calculate, say, margin of error?

So what does it mean to say that, in a tightly controlled experiment, I improved the state of my knees by doing x and y, when I was the only subject in the experiment?

Again, statistically: not much.

But there’s another way to look at this.

And that’s by considering the likelihood that a given experiment can actually prove cause and effect.

See, it doesn’t matter if you have 100,000 subjects in a study, if that study is poorly designed and/or poorly executed, making it impossible to isolate cause and effect with a high degree of confidence. The study’s researchers may think that they have reached a conclusion that A leads to B (or doesn’t lead to B), but this is a dangerous sort of illusion if the study is fundamentally flawed. Long-term knee studies of the type, “Activity X is good/bad for knees” are especially prone to this problem. In Saving My Knees, I take a close look at one.

I won’t retrace that ground here. But consider an example to help illustrate what I mean: Suppose you run a one-year study in Smallville to see if daily swimming helps really bad knees. You believe that to be true, but at the end of the trial, you’re puzzled to find the swimming group actually does worse than the control group -- in a statistically significant way too. You publish the results.

“Swimming adversely affects knee joint health,” you proclaim.

But -- what if you discover the Smallville swimming pool is on the sixth floor of a building that has no elevator? And for the subjects in the study, who have really bad knees, climbing all those steps is too much for their joints?

This is the kind of problem you face in a study when important variables aren’t tightly controlled. Granted, my example is a bit extreme and far-fetched, but what’s undeniable is that, in studies that attempt to show “Activity X helps/hurts bad knees,” for about 98 percent of the time (during waking hours), the knees of the subjects will NOT be engaged in Activity X. And what they’re doing during that 98 percent of the time -- whether it’s ill advisedly climbing six flights of stairs or kneeling to scrub a floor or running to catch a public bus -- can be vastly significant and skew the results in a big way.

There are other flaws of long-term knee studies that attempt to show causal relationships, which are independent of sample size. In Chapter 8 of Saving My Knees, I mention some more.

My experiment, on the other hand, was closely observed and very well-controlled when compared to a typical study into what helps/hurts knees. Now you can argue that doesn’t matter -- that I have the constitution of a space alien, or that my patellofemoral pain syndrome was wholly unlike anyone else’s, so what I did to cure my bad knees won’t help others.

Of course, I would disagree. Further, I’d argue that sometimes you can learn a lot more from a one-person experiment that’s very well-conducted than a multi-thousand subject trial that isn’t.

So maybe it’s not so bad after all being a sample size of one.