Quality, quantity, and academic promotion

After my last detail-oriented post, I thought I'd write about something more abstract; on a day like today, something like coincidences or binary arithmetic might be appropriate. But apropos to the topic actually at hand, life goes on pretty much as usual here, both today and tomorrow. Academia is infamous for its Byzantine promotion criteria, and it's striking how many of them boil down to quantity versus quality.

Grants, for example, are easy: quantity. Everybody knows that funding rates are low, but what does that imply about funding strategies? To draw a scientific parallel, consider the problem of biomarker discovery in high-dimensional data. To whit, suppose you're studying a rare disease, and you've managed to locate about 10 patients who are suffering from the disease of interest. You round up 10 or 20 or 100 healthy control subjects, and you make hundreds of measurements of the entire population: height, weight, blood pressure, temperature, you name it. Your goal is to find a quick and easy diagnostic test that will differentiate patients with the disease from healthy subjects. The upshot is that in a case like this, one of the somewhat counterintuitive findings of recent work in high-dimensional statistics is that there may be no "one right answer." Your disease is rare; you only have ten patients to study. Given that it's a severe enough disease to warrant study, it's likely to be fairly extreme, and most of the variables you've measured will probably be good indicators. All ten patients have a fever; all of them have high blood pressure; all of them have unusually low BMI. Using any one of these variables or any few in combination would be equally good, because you have so many to choose from relative to your number of patients. This is the heartbreak of gene expression biomarker discovery and genome-wide association studies - if you're looking for a set of genes on which to blame a disease, your problem is most often that you have too many good candidates, not too few. Which measurements/genes/what-have-you actually end up being used in the clinic as a diagnostic tool is then essentially up to chance, since so many of them are (nearly) equally good.

The analogy to grantwriting is obvious, as is the rather distressing winning strategy in such a situation. From a pool of n submissions, the top m are indistinguishably good (within experimental error). If k much less than m are selected, the selection will happen essentially by chance. In other words, beyond a certain point, winning or losing the game is almost random - it only matters how often you play. How's that for depressing? And although it's been obvious to someone who studies biomarker discovery for years, it's bad enough now that that Nature article I linked above comes to exactly the same conclusion. Consider what that means for junior faculty, who have no reliable income source for the lab yet: quantity over quality.

Publications, thankfully, fall squarely in the opposite corner: quality over quantity, with a caveat I'll speak to below. There are several ways to arrive at this conclusion, common sense being an easy one (for once). As a quirky PLoS CB article put it, for all academia's ivory tower reputation, modern science relies a lot on "star power." Your ability to succeed is often judged based on intuition and perception, and you'll be remembered for your one big hit, not for the many reasonably solid ones (although occasionally for a big flop, too). I've seen this pointed out in a context that I frustratingly can't find a link for (easier to stalk Arnold than academic strategy). Specifically, one of the reasons big labs tend to succeed is that it only takes one "productive" member to make the whole group look good, so in that way it again becomes a numbers game - just of people, not papers. Everybody will know you for that one big NSC paper; nobody will know you for your ten equally great papers in low-impact journals (with some prominent exceptions).

The one very important area in which this rule is broken is in mentorship. It's important for obvious reasons to give a student or postdoc a project they can take ownership of and publish somewhere in a reasonable amount of time, whether or not it will fit into the grander scheme of science. Inasmuch as the big three publish incredibly specialty biology sometimes in areas I couldn't care less about, it does make me smile because it highlights the fact that individual projects performed with care and an eye toward quality can still be presented in a high-impact setting. But, again, it's a coin flip - and the fact that most individual projects end up elsewhere shouldn't be a reason for them not to exist. Quantity matters in publications when it means that each lab member can champion a project he or she is interested in, and for once that shouldn't preclude quality as well.

Thankfully, while education is still included as a criterion for academic promotion and is statistically significantly associated with success, it's debatable whether that association carries a particularly large effect size. I've had at least one friend leave academia for the explicit reason that he felt pressured to intentionally teach poorly - to put as little time and effort into education as possible. Quality teaching takes a tremendous amount of time, so there's not really much of a tradeoff to make here. It's both quality and quantity or bust. I'm in no position yet to speak with any authority on the real place of teaching in the promotion process, but I'll pose a challenge instead. Make your first lesson plan covering, say, a month's worth of classes: lectures, readings, homework assignments. Do a good job; work as hard and spend as much time as it takes, no matter how much you hate it (I understand that not everyone enjoys teaching!) Then come back and consider how much it's improved your communication, writing, presentation, and organization skills. I'd say that although quality might, sadly, not always matter for education in modern academia, it matters at least once. You'll learn more in that first wholehearted effort than your students will.

Then you can slack off to make your tenure committee happy, right?

To end on a positive (and related) note, I found out recently that I've been both quantitatively and qualitatively shown up by a friend of mine at the Broad who writes quite a bit more regularly than I possibly could here. His content's somewhat more wide-ranging - I don't know nearly as much as him about politics and education reform - but he seems to be keeping a good eye on scientific happenings around Boston and the human microbiome community. Definitely recommended reading!