Open Peer Review

The scientific journal Nature is conducting a fascinating experiment in "open peer review", which it describes this way:

In Nature's peer review trial, lasting for three months, authors can choose to have their submissions posted on a preprint server for open comments, in parallel with the conventional peer review process. Anyone in the field may then post comments, provided they are prepared to identify themselves. Once the usual confidential peer review process is complete, the public 'open peer review' process will be closed. Nature will report on the results after the trial period is over.

In parallel, it's conducting a online debate about the future of scientific peer review in the web age, and has asked various scientists and writers to contribute their thoughts. Mine was on how the online definition of "peer" (basically anyone, as in "peer-production" and "peer-to-peer") can help us rethink the academic definition of "peer" (usually a tenured PhD professor) for the purpose of identifying the most important scientific work. Here's an excerpt:

The free online Public Library of Science (PLoS) journals are planning to extend the open peer review model by adopting conventions from the blogosphere: an open comment area for each paper, 'trackbacks' that show which sites are linking to it, and perhaps a reader ratings scheme. Michael Eisen, a genomics researcher at the Lawrence Berkeley National Laboratory and one of PloS's founders, says the hope is to capture some of the post-publication wisdom already found in academia, but rarely accessible to others.

PLoS still uses expert researchers to review papers before publication, but the editors realize that these scientists often have little time to really dig into a paper. By contrast, readers of a paper after publication may also have an opinion, and many (especially graduate students) have the time to evaluate the paper in depth. The online environment means there’s no reason not to record it.

Such a record would have the effect not only of continuing peer review after publication, but also of making it easier to find important work in a blizzard of papers – they’re the ones that are being buzzed about. It is also easier to ignore poor work that slipped through peer review – these are the papers with the withering comments and poor ratings.

Best of all, such an open peer-review process taps into something that already exists: journal clubs. Every day, thousands of researchers and students are discussing the latest papers, but their insights and opinions are not recorded and shared widely. This information needs only to be collected, organized and distributed to become far more useful. It's now possible to tap such collective intelligence online by doing to scientific publishing what the web has already done to mainstream media: democratizing it.

You can read the whole thing here and comment on it here.

Comments

[Bill Barth]

I don't think the usual definition of "peer" as exclusive to "tenured," "PhD," or "professor." It's certainly not the case in the engineering and mathematics disciplines I'm familiar with. I've been doing peer review of journal articles since I was a senior PhD candidate, and I still do them as a (PhDed) research associate (similar in rank to an untenured professor) at a major research university. Journal editors are responsible for choosing the most knowledgable reviewers they can get, but I don't see any reason that should be limited to tenured professors. Several journal editors I know are more than happy to have qualified senior graduate students, untenured junior professors, and other types of researchers (say the PhDs that work at the national labs) amongst the reviewers for their articles.

[Maxine Clarke]

Thank you for the great posting, Chris.

[Chris Anderson]

Bill,

I agree that it shouldn't be the case the peer review is limited to professors and their PhD equivalents in industry, but at least for the premier journals that's usually the case. These are usually the individuals whose research history is long enough for them to have established a reputation in a field, by which I partly mean that the journal editor knows them and they've peer reviewed before. You're right that a more inclusive approach is better, and that's what my essay attempts to explore.

Chris

[David Smith]

Interesting article Chris.

As someone in the STM publishing industry, I offer the following thoughts. I was involved in some experiments on post peer-review commenting waaay back in the late 90s/early2ks. Back then we had forums associated with each published article and it was very easy for our journal readers to comment on the published work. For some reason it never took off. The facility was simply never used (except at holiday times when we used to publish funny articles which got lots of activity). Based on this, and some other investigations I did, I came to the (not original) conclusion that we were dealing with a population that consisted mostly of lurkers. There is also the fact that most papers are read by a very small number of scientists who generally know each other (or at least are pretty familiar with each others work). This changes the dynamic I think. To avoid hogging the comment space, I'll try and boil my argument down to this...

In the scientific community at least, the combination of small population sizes (per individual paper), coupled with the tendency to lurk, coupled to a reticence by those lower down the scientific pecking order to be seen to criticise those who are higher up, tends to act against such post peer-review tools. There's a reason why we consistently find that all parties like to continue anonymous peer-review and I think it applies equally to this kind of activity as well.

Should you want to discuss further - drop me a line.

David

[Sean]

I am in no way a scientific community member, but I wonder if the phenomenom described by david may be similar to that of the "paperless office."

What we've found in my large governmental organization is that the paperless office went over very poorly when people first started to attempt to create the paperless office, the workers of the time revolted. The mechanisms involved were clumsy, often poor digital replications of the paper process. There have been numerous articles referencing the need of people to work with a physical piece of paper when reading or doing edits.

What we've found in my office is that as older workers retire and younger people enter the market place, as well as technology and process improvements are brought online, the paperless office is more and more actually becoming the reality. This has been aided by dual screened computers allowing the direct comparision of documents, user comfort with reading things on a monitor as opposed to a book, better engineered processes that properly take advantage of the digital medium, etc.

My point, WRT to the question at hand, is that as more people who are use to their ability to comment anywhere, on anything, and be judged solely on the value of their argument (as I do here), you will see the lurkers and those afraid to comment on the work of the people higher in the hiearchy slowly begin to degrade. Its a matter of familiarity, habit, and culture.

[Camera Bags]

I'm afraid y'all have gotten ahead of yourselves. You've introduced a primitive tribe to the Roomba, and can't understand why they haven't all jumped on the robotic vacuum bandwagon, instead preferring to use green branches, which do a much poorer job on their floors.