5/14/2007

The Internet and Me and Complete Harmony

Yesterday, Bossman asked me the following question: "Do you think that the ACS will eventually host a reddit-type scoring of it's publications?  Do you think it's a good idea?  Nature's experiment with open peer review failed, but this approach may have more appeal." As he put it, despite my inexperience with peer review for major publications, he wanted the opinion of someone "embedded in the internet." And embedded I am. Now I'd be remiss if I didn't say that a lot of my opinions were shaped by reading Paul's blag and the rather prolific commenters, especially on this post. Below lies my response.

I constantly waver back and forth on this issue.

I'm not familiar with Nature's open peer review experiment, and since I've never been a part of the peer review process for any publication,my opinions are based simply on what I think seems the most reasonable approach. I don't see ACS adopting a reddit-style scoring system anytime soon, and I think one of the biggest obstacles facing a more open peer review process is clout. ACS journals are respected, in part, because of the way they go about their peer review process. It's a tried-and-true technique. However, we've seen the limits of the success of this peer review process lately, in particular with the serial retractions of Dalibor Sames' papers. Some feel that the retraction process was handled smoothly, but others believe that if things were a little different, then the issues could have been handled much more quickly. It took nearly four years for some of the papers to be retracted by Sames. If the peer review process was more open (more "democratic," I suppose), so say the open review advocates, this scandal could have been taken care of in a much more quick and efficient manner, and it would prevent these things from happening as often. Right?

Well, yes and no. Although I'm of the "internet generation" and pretty much everything I do revolves around it in some way, I think that exposure has made me even more suspicious of the advocacy of open review. On places like reddit and digg (especially digg), one senses the tyrrany of the majority implicit in these communities. Anyone whose opinions differ from the majority are quickly slienced, it seems. To be fair, those who are silenced probably had it coming (poor arguments, foul language, etc.), and regardless of what you think of such a democratic method (I abhor it, personally), places like digg and reddit are talking about trivial matters. We are talking about professional science journals. And is that what the future of scientific publishing should be? Tyrrany of the majority? Absolutely not. In order for a reddit-type system to work, there would have to be restrictions put on who is allowed to review and comment on papers. For example, JACS papers could only be commented upon by ACS members, and no anonymity would be allowed- you make a comment, people will know who you are and what you said. I think, if the open review process is going to work, it has to work within a community that respects its peers- transparency is essential. The internet is full of assholes who ruin communities, and that REALLY needs to be taken into account.

Let me present one example that addresses another issue I have with an open review process. Let's say Prof. X publishes a paper on Topic A. It gets accepted into JACS and the Topic A paper online becomes a forum on Topic A, as all of these types of paper-based forums will become. The relevance of Topic A will be addressed, Prof. X will answer questions, defend his work, argue with Prof. Y who is also working on Topic A, etc. That would certainly be a paradigm shift- Prof. X and company will now have to actively defend their work on the internet. Of course, you could have Grad Student Z just do it for you, but get a bunch of young professors who incorporate such work into their daily lives, and the people will expect more author AND prof dialogue. I think older professors, weaned on the older methods, may be alienated from such a method. It would be a major shift in how things are done, and I don't expect a lot of the older crowd to be able to accomodate. There is also the issue of reddit-style popularity and how notoriety may affect neutrality: popular articles may be popular only because of who wrote them, and it might be difficult for new PIs to get the kind of new notoriety that they need to establish themselves. As it stands, a paper by Marty Burke can stand next to one by EJ Corey as potential equals. A reddit-style "popularity" system may keep such an equality of oppourtunity from occuring.

Sorry, that was long, but in conclusion: I don't expect a HUGE shift in how ACS journals are reviewed any time soon, but I think it would help transparency in review, which in turn can aid in scientific dialogue- the internet has made everything faster, so why not dialogue in research as well? However, such an implementation, if a journal is to stay respectable, will have to impose limits on how dialogue is performed. I can see great things happening from it, but with great power comes great responsibility.

11 comments:

jokerine said...

I think the biggest problem with open article discusion is the dictatorship of the majority you mention. That is on thing wikipedia still has to work around. Also, there will be Trolls.

Uncle Al said...

Peer review excludes both abject stupidity and eldrich brilliance. Peer review enforces mediocrity. Science as a whole does best when there is perceptible inertia. However...

"Greater than the death of flesh is the death of hope, the death of dreams. Against this peril we can never surrender. The future is all around us, waiting, in moments of transition, to be born in moments of revelation. No one knows the shape of that future or where it will take us. We know only that it is always born in pain." G'Kar, Babylon 5

If you want to score a field goal, somebody has to kick the ball. That somebody is definitely not the Coach.

Paul said...

What's cool is that the JACS comment system is going to happen when one of us gets some free time. You register commentonJACS.com, do a little PHP/SQL coding, then all of a sudden people can come to your site and comment on JACS articles and have their comments persist for eternity. Of course, without "official" support, you're going to attract a different crowd at the beginning (the unwashed Internet types), but the system would be so useful that it would take off eventually. A comment system *will* happen, it's only a question of who and when.

Excimer said...

Paul: hmm... you're advocating an outside comments board, while Bossman is advocating an official ACS one, then? I hadn't thought of that. Maybe having both would be interesting..

sam said...

I think that some type of tagging and voting will eventually be helpful to sorting and finding scientific journal articles. But it would need to be incorporated into a larger system of finding articles---it could be one of many parts of a searching algorithm.

baoilleach said...

What are you guys talking about? Don't you already realise that your blogs are already providing 'unofficial' reviews of papers. Thanks to the Greasemonkey script described here your comments are already plastered over the Table of Contents of JACS, etc.

It's not waiting to happen - it's already here!

David Eaton said...

I hear a lot of stuff about peer review being broken, and it has some pretty glaring problems. I imagine it will have to evolve, and I'd be surprised if it didn't involve more openness and the application of technology, but I think it's going to require doing a bunch of dumb things, over several iterations, to get something significantly better than what we have.

Do any of you guys pay any attention to the arXiv.org papers? I try to scan the soft condensed matter and materials papers, but I admit most of it is a bit over my gourd theoretically. I still get enough out of them to keep going back.

The papers are pre-pre-prints, not peer reviewed, but I like seeing them early. If you follow the archives for a while, you will see papers evolve. I know that it can be pretty brutal with that many eyes on the work.

Most of the stuff gets to the lit eventually, though some pretty significant stuff never has. The Poincare' conjecture 'proof', for instance.

The interwebs sort of feeds the 'stick-it-to-the-man' countercultural vibe I think thrives around physicists. Not that chemists are exactly bastions of cultural rectitude (whatever the hell that means) but I think chemists have been less radical in using technology to keep bleeding edge stuff out in view. I figure it will change, though.

Paul said...

I am actually an advocate for an official system, but my point is that if the editors don't wise up and create one, then someone else is going to do it by himself because it's a good idea. In the latter case, the editors will have lost a good deal of control and will have to make an improved system to pay for the switching cost.

And that greasemonkey script looks awesome. I have been meaning to blog about it for some time.

baoilleach said...

Following on from Paul's initial comment, I've posted a technical discussion of a framework for off-site journal commenting here.

Mitch said...

It would be rediculously trivial to implement. I have most of the code already written that would just grab the new articles when they show up on the rss feed. After very informal discussions with ACS people, they very quickly and rightly frowned on anyone doing this.

Mitch

baoilleach said...

I've implemented a simple system to vote for your favourite papers. It's described here.