Why Many Knee Studies Are Seriously Flawed

I’ve cited many knee studies on this blog, usually to buttress some point I’m making.

But I’ve also come to the conclusion that a certain kind of knee study has to be approached with much caution. That’s because it suffers from serious and intrinsic flaws.

First, let me be clear what I’m not talking about: so-called “natural” studies that, instead of testing a hypothesis, look at say how knee cartilage changes over a two-year period, and how characteristics of subjects such as age and BMI influence those changes. (Fortunately, one of my favorite studies falls in this category: the two-year study by Australian researchers showing that, in a whopping 37 percent of subjects, a cartilage defect actually improved somewhere in their knee.)

The kind of study I'm talking about is more ambitious. It attempts to prove a link between some activity X and the health of your knees. And it suffers from serious limitations.

To see the issues, let’s create a fictitious study that has a sensible premise: it sets out to prove that walking benefits bad knees.

Researchers round up 600 subjects with knee pain and split them into three groups. Group A doesn’t walk at all for exercise, Group B walks 1-7 miles a week (as much as a mile a day), Group C walks more than 7 miles. And, to better capture long-term changes, the researchers decide to observe the subjects’ knee joints at the beginning and end of a 10-year period.

Okay, why is this study already in trouble, just by design?

(1) There’s no way to account for “category drift.”

Lives can change a lot over 10 years. A moderate walker may become a super walker -- or a nonwalker. Or someone may jump back and forth between these three categories quite a bit.

What if a moderate walker for nine years becomes a nonwalker in year 10 -- which happens to be the same year he’s surveyed by our researchers about his level of physical activity? If his joint health turns out to have improved, the results would be recorded under “Being a nonwalker is good for your knees.”

That makes no sense, because for 90 percent of the study’s duration he was a moderate walker!

(2) The big problem: Exercise is only a small part of how someone uses their knees in daily activities.

Consider this math: Let’s say you’re awake 16 hours a day. Let’s say you walk a mile a day for exercise and that takes 20 minutes (that’s a three-mile-an-hour pace -- not too demanding). So if you do the math, how much of each waking day are you exercising by walking?

Barely 2 percent.

Which raises the question -- what are your knees doing the other 98 percent of the time? This matters hugely! Because if "walking for exercise" is the variable that is being studied, then what's going on during the other 98 percent of the day is going to contribute to “variable pollution," contaminating the findings.

(Question to ponder: If subject A has a job where he walks several miles at work, but doesn’t walk for exercise, while subject B is deskbound but walks a mile a day for exercise -- who’s really doing more walking?)

(3) The study never makes adjustments for what level of activity (and type of activity) is appropriate for each particular subject.

Say Mary walks one mile a day, which puts her in the “moderate activity” category. At the end of the 10-year period, an MRI reveals her knees have gotten worse. So should her example be used to support the thesis that “moderate walking is harmful for people with bad knees”?

Not necessarily.

Maybe Mary’s joints are so bad that she needs less walking, or shorter bursts of walking, until she can strengthen them. Maybe, had she been in a group that walked only one-quarter mile daily, while taking 60 steps around her room every half hour, her knees would have gotten better.

It’s sort of useless to create a study to draw conclusions about how much of activity X is beneficial when you have no idea how much of activity X each particular subject should be doing (which depends on the strength of their particular joints).

Okay, those are some reasons why I consider many knee studies badly flawed.

Now it’s time for a visit from my imaginary critic:

Great, so researchers shouldn’t undertake a study unless it’s perfect? Subjects’ environments must be totally controlled for all variables? That’s ridiculous. You can’t find any study that meets those high standards. Basically, you’re throwing a lot of good science out the window.

My response: Yes, the perfect is the enemy of the good.

But showing how the good can be flawed -- and sometimes, not be very good at all -- isn’t meant to be an exercise in nihilistic nit-picking. Recognizing that serious flaws exist for many knee studies, and knowing what they are, means you can better evaluate to what degree the study you’re looking at may have escaped those problems.

For instance, category drift will be a bigger issue with a longer term study that samples a subject’s activities at only one point in time. So that may argue for the superiority of a shorter term study (which has its own drawbacks, true).

My message here, once again, is about thinking critically. There’s so much bad and suspect information about healing chronic knee pain that we’re foolish if we don’t think critically.