Another piece of academic logistics that's been of interest in the lab lately has been how to effectively referee a manuscript. On the surface, this is a relatively straightforward process: a journal emails you, you reluctantly hit "Agree", and two weeks plus one day later when you receive an automated email reminding you that you're late, you hastily dash off a few sentences deriding the document as the worst piece of drivel to ever disgrace your inbox. Well, at least that's the way it goes if you happen to be the third reviewer.

There's sometimes a bit more to the process. First, consider that your job in reviewing a manuscript is to improve the scientific literature and community. This is a superset of the nominal task of determining whether the text itself is sound, or even whether the project is a good fit for the proposed journal. I'd say that refereeing a manuscript actually comprises three tasks, only one of which is tied to the actual document:

• First, you do want to establish whether or not the paper's any good. We'll unpack the details of this task in a bit.
• Second, you want to convince the editor that he or she should agree with you. Performing a review doesn't just mean stating your opinion; it necessitates convincing someone else to do what you want, which is a bit different.
• Third, you want to engage in as productive a dialog with the authors as possible, and to a small degree the other reviewers as well.

Thus every review is primarily a conversation between you, the editor, and the manuscript's authors, with some additional dynamics provided by the (typically two to three) other referees. My characterization of the default review process would be:

1. Authors submit manuscript, recommending three to five possible reviewers in the process. Much self-policing occurs, such that the journal is likely (but of course not guaranteed) to be an appropriate match.
2. Journal editor chooses two to four reviewers, typically all but one of whom come from the authors' recommendations.
3. You, as a reviewer, critique the manuscript, determining the soundness of the science, the fit to the target journal, and the clarity of its presentation.
4. Upon receiving all reviews, the editor decides whether the paper should be accepted, rejected, or modified; one of the latter two is almost always the case.
5. The authors revise the manuscript and resubmit.
6. The editor can decide whether or not to send this revision out for re-review; there is more variability here, but re-review with the same referees is still the most common case.
7. Second round reviews are typically much shorter, either basically agreeing that the paper's publishable or that it isn't.
8. Finally, the editor either rejects the paper (rare at this stage) or begins the publication process with the journal.

With this in mind, we'll expand some of these points in a bit. First, I'd like to emphasize that all of this is a conversation. Civility is key, and no matter what you write or what format it's in, you should envisage yourself saying it out loud to the manuscript's authors, in the presence of the editor. Over time, many of them will be people you know, and even when this isn't the case, you will find that the late stages of this process are vastly less awkward if you've been a rational human being. Even more critically, the more papers you submit yourself, the more the Golden Rule will apply. Remember how mad you were the last time you received an insensitive, unreasonable review from someone who barely read the paper?

So, I've promised repeatedly that we'd discuss the review process itself. I typically structure a review with four parts:

1. The first paragraph is a summary of the manuscript. What did the authors do, how, and what did they find? The goal is to show that you read and understood the paper; if there were any major misunderstandings, the authors can point them out in the response. Always be ready to admit that you could as easily be at fault as they!
2. The second paragraph is a summary of the rest of your review. What were the 1-2 best features of the study, plus the 1-2 worst? Are there any points you feel the authors absolutely must address in a revision, or, heaven forbid, things that you particularly liked? Don't be afraid to make comments about good things in addition to bad ones!
3. The third section should list major points for the authors to address in revision. These should be high-level structural or scientific issues, and their descriptions here can be fairly extensive. List them in order from most to least important, although everything in the "major" section should be fairly important - no typos here! Some examples of what I might call out as major points include:
   • Missing experiments or evaluations (unless they're doing something so novel nobody's ever tried it before, I want to see a quantitative comparison with previous work).
   • Fundamental misinterpretations of results; are the authors overlooking potential major sources of bias or artefactual behavior? Could noise or systematic bias produce the same result as legitimate science?
   • Implementations and/or source code. There's no excuse to be publishing computational results without making an implementation available, preferably open source.
   • Major comprehensibility or language issues. Again, no typos, but if the paper is too structurally confusing, wordy, or typo-ridden for you to easily understand, say so. Remember that you're reviewing it because you're an expert - if you don't get it, no one will!
   • Scientific or factual flaws. Hopefully this one doesn't happen very often, but I've seen the occasional manuscript touting things like an AUC of 0.3 representing superlative performance.
4. The fourth section is similarly moderate points. This is most often where I ask questions of the authors (e.g. "Did you explore X?" or "Wouldn't it be interesting to include Y?") and where I suggest optional modifications to the paper.
5. Finally, minor items should be last, and these often consist of typos and non-scientific issues of comparably low impact.

This sounds like a lot - and it can be! Again, remember what it feels like when you get a crummy review back after submitting your hard-won completed manuscript? Yeah, don't do that to anyone else. If you agree to perform a review, make sure you do understand the work being presented (if at all possible) and that you spend the time to convey your thoughts about improving it to the authors cogently. On the other hand, reviews for normally sized papers (e.g. 6-8 pages) shouldn't take more than a few hours at most. If there are issues you don't understand, don't burn a lot of time unraveling them - say so and move on. If you don't get it, no one will.

Also, as an aside, don't be afraid to suggest taking material out or moving it to the supplement. Once more, your goal is to improve the field, not to take the manuscript at face value. It could be perfectly valid and interesting science, but if it's incomprehensible (or too long and boring), nobody will bother to read it! Supporting figures that aren't necessary to understand (and believe) the science, speculative figures that could be differently interpreted, or detailed methods that aren't needed to reproduce the result are all great candidates for supplemental materials and text. The supplement in general is an under-appreciated tool these days, particularly now that one can attach essentially any electronic file you'd like (think raw data!) to a manuscript online.

I'd like to end with a quick example, for which (after some deliberation) I'll call out a paper by my past self, not least because it's short.

In this manuscript, Huttenhower et al present Sleipnir, a C++ library for large-scale genomic data processing. The library includes a variety of tools and data types for rapidly manipulating various experimental data, particularly expression arrays, and for performing simple machine learning tasks using these data. The authors demonstrate Sleipnir's use to obtain, integrate, and make inferences from a set of ~200 yeast experimental datasets, a task taking ~four days on a workstation-class machine.

First, I appreciate the fact that the authors have provided extensive documentation and source code with the Sleipnir software, which is freely available online. The authors also make a compelling case for Sleipnir's efficiency, although it is likely that professional software could perform e.g. the tasks in Figure 1 in substantially less time. Although I appreciate the manuscript as a whole, I have several remaining concerns as detailed below:

Major
----
* Although I appreciate the space limits of an application note, it would be much easier to understand the benefits of the software if provided with a few additional examples of tasks that it can be used to perform.

Moderate
----
* The authors focus on applications of Sleipnir to tasks that are heavily customized to their particular lab. It is not clear to me whether other groups would ever be interested in carrying out the large integration tasks for which Sleipnir is most optimized. I would be reassured by seeing additional evidence of Sleipnir's use outside the authors' own group.

* Perhaps the authors could add a supplement including one of the tutorial examples present on the Sleipnir web site?

Minor
----
* The URL provided for Sleipnir seems to redirect to a secondary server of some sort?

* Sleipnir's build process seems to be exceptionally complicated and almost-but-not-quite follows the Linux autotools standard; the relatively large number of nonstandard external dependencies makes it difficult to build with full functionality.

* The authors might consider directing readers to Microsoft's Visual Studio Express tools to allow building on Windows.

After having written and used Sleipnir for almost five years now, let me tell you that's not an easy review to write! Nevertheless, suppose the editor decides to let the authors respond, which they do (a topic to be discussed in a later weekly installment), and a beautifully revised manuscript again finds its way into your hands for re-review:

The authors have addressed essentially all of my previous concerns, and their revisions have substantially improved the manuscript. I have no further comments, save for the minor suggestion that I would again appreciate seeing additional examples of how Sleipnir can be used for someone _else's_ investigations added to the manuscript or to a supplement.

That's not an atypical scope for re-review; I told you they were often very short! Inasmuch as a first review can say nearly anything, the goal of a second review is not to request additional work or changes - if you were going to do that, you should have done it the first time. The only three messages I typically try to convey with a re-review are either A) nice work, you're done; B) you and/or the editor ignored my previous suggestions, and I think the paper shouldn't be published; or C) I like the paper now, but you should really fix these last couple of problems before publication.

In closing, keep in mind that manuscript reviews really do serve as a gatekeeper. Inasmuch as there are authors who will perfectionistically never release a manuscript for submission until it's been polished beyond the point of reasonability, there are likewise authors who will submit demonstrably incomplete writeups, expecting to finishing them up during revision. Just like we all have a boss to keep us on track at work, the review process provides some minimal oversight and accountability - not in a bad way, but as a straightforward check to the mistakes we make as busy people with limited time and resources. In responses to reviews, we almost always thank the reviewers for "helping to improve the manuscript" or some such, and inasmuch as it's a cliche, it should also in the best case be the truth: a goal of the review process is to take some of the burden of perfection off the authors' shoulders. What comes out of the song and dance that is scientific review should be a document that actually does enlighten the reader and serve as a record of a job well done for everyone involved.