Digital Humanities


King's College London, 3rd - 6th July 2010

[Image: KCL Photo Collage]
[Image: London Photo Collage (Somerset House; Globe Theatre; Millennium Bridge; Tate Modern)]

Open vs. Closed: Changing the Culture of Peer Review

See Abstract in PDF, XML, or in the Programme

Fitzpatrick, Kathleen
Pomona College, USA

Despite our differing research methodologies, subjects, and motives, the one thing scholars across the disciplines and around the world might agree upon is the significance of peer review. Peer review may be the sine qua non of academic work; we use it in almost everything we do, including grant and fellowship applications, hiring and promotion processes, and, of course, in vetting scholarly work for publication. We all operate under the agreement that peer review is a good thing, by and large, both a means of helping scholars improve their work and a system for filtering that work for the benefit of other scholars.

However, as I argue in Planned Obsolescence, the means by which we conduct peer review demand careful reconsideration and revision as academic publishing moves increasingly online. Clay Shirky has argued that the structures of the internet demand a “publish, then filter” process, encouraging the open communication of all of the results of scholarly investigation, followed by a process that filters those results for quality (Shirky 2008). I explore the reasons such a transformation is desirable at length in Planned Obsolescence, primarily that it makes little sense to replicate a mode of review designed for print’s economics of scarcity within the internet’s economics of abundance (see Jensen 2007); if what is scarce in the age of the network is not the means of production but the time and attention available for consumption, the best use of peer review would be to help researchers find the right text, of the right authority, at the right time.

A born-digital system of review would work with rather than against the strengths and values of the network by privileging the open over the closed, and by understanding the results of peer review as a form of metadata enabling scholars to find and engage with research in their fields. How to build and implement such a system, however, remains in question: how do we devise a networked review system that is open, honest, and thorough, that draws the best from the “wisdom of the crowds” (Surowiecki 2004; Anderson 2006) while upholding the standards that review is meant to serve?

Several examples of online review processes already exist; within humanities journal publishing, the most significant may be that of electronic book review; articles submitted there are posted in a password-protected review space, where registered users of the site can read them, leave glosses, and recommend acceptance or rejection. However, though the editors use my term “peer-to-peer review” in describing their system, it falls a bit short of the truly open system I imagine; the review system is still kept behind the scenes, and while the reviews are crowd-sourced, the reviewers producing them aren’t asked to take responsibility for their opinions by expressing and defending them in public. This aspect of peer-to-peer review is key; just as the quality of the algorithm determines the quality of a computational filtering system, the quality of the reviewers will determine the quality of a human filtering system. Online peer review must made open and public not just as a means of increasing communication but as a means of increasing reviewer accountability, providing for the ongoing review not just of texts but of reviewers.

In order to experiment with the possibilities for an open review system, and with the consent of NYU Press and my editor, Eric Zinner, I placed the entirety of Planned Obsolescence online in late September 2009. The text was published in CommentPress, a WordPress plugin developed by the Institute for the Future of the Book, which enables the discussion of texts at a range of levels of granularity, from the paragraph to the page to the document as a whole. At the same time, NYU Press sent the project out for traditional peer review.

Such experiments have been conducted before; in 2008, Noah Wardrip-Fruin published a draft of Expressive Processing through Grand Text Auto, while MIT Press sent it to outside readers. Noah, however, wasn’t seeking to create a head-to-head contest between closed and open review; he was motivated by the desire for feedback from a community he trusted (see Wardrip-Fruin 2009b). My motives were a bit more complex; I wanted that same community-based feedback, but I also wanted to test open review against more traditional reviews, to gauge differences in the kind and quality of responses produced within an online system, and to project the kinds of changes to CommentPress that might help transform the plugin into a viable mechanism for peer-to-peer review.

In slightly less than six months, Planned Obsolescence received 205 comments from 39 different readers (not counting my own 78 responses). These comments are by and large excellent, and have been extremely helpful in thinking about the revision of the manuscript. Most of the comments, however, are locally oriented; CommentPress’s paragraph-level commenting strategy encourages a close attention to the particulars of a text rather than a more holistic discussion. This focus on the text’s details in the comments wasn’t unexpected; we anticipated that the traditional reviews, being written after the entire manuscript had been read, would tend to focus a bit more on the big picture than would comments made in medias res. This assumption did largely bear out; the offline reviewers tended more toward an assessment of the overall argument.

Our first tentative conclusion, then, was that a functional open review system would require clearer ways for online reviewers to leave broader comments. An update to the CommentPress plugin, released a couple of months into our experiment, helped provide that functionality by highlighting the “community blog” section of the site, which in theory would allow members of a community of readers to engage one another in discussion of their reviews and of the project as a whole. In actual practice, however, that engagement did not occur, though it remains unclear whether this is due to the blog’s belated introduction or some other issue.

Additionally, however, Zinner asked the offline reviewers whether they would be willing to participate in our process, allowing us to post their reviews for discussion and response; one, Lisa Spiro, agreed. Spiro’s willingness to participate, and the generosity of her review, revealed the importance of the social commitments involved in the peer review process. Those scholars who have long undertaken the often thankless work of peer review have largely done so out of a commitment to the advancement of knowledge in the field. But fostering participation in online discussion requires not just intellectual interest on the part of individuals but also a solid, committed social network. Reviewers participating in an open process must have a stake in that process beyond that of the disinterested reader; they must understand the text and its author to be part of a community in which they are invested and to which they are accountable.

Beginning in March 2010, MediaCommons will conduct another open review experiment, publishing a small group of papers being considered for a special issue of Shakespeare Quarterly. Through this experiment, we hope to explore a number of variables: the relative weights of commitment to subject matter and commitment to digital methodologies in determining participation in open review, the level of engagement in the review of article-length (as opposed to book-length) texts online, and the structures of participation in the review of work by multiple authors in one venue.

Both experiments involve the review of comparatively traditional forms of scholarship, the book and the journal article, which we have opted to begin with for two reasons: first, that transforming the processes of reviewing these forms of scholarship presents the broadest potential impact on academic publishing as it exists today, and second, that it confines the question under consideration to mode of review, rather than expanding into criteria for review. That last is extremely important; many, if not most, scholars working in new forms of multimodal scholarship have encountered the sense that the academy in general does not know how to review such work. We hope to experiment in the future with models for review of new forms of scholarship.

This paper will, in the end, argue that a truly effective peer-to-peer review system will need to place its emphasis not just on developing the technological network but on developing the social network; it must be focused around clusters of scholars who are already in dialogue with one another. It must also be accompanied by a shift in values that encourages scholars to understand the business of reviewing as being a commitment not just to the advancement of intellectual thought but to the structure of that community and its dialogue. And, as participation in such review requires significant time and energy from a larger number of scholars than traditional review, the academy must recognize the importance of reviewing, acknowledging the significant labor involved, creating structures through which reviewers can receive “credit” for their work.

MediaCommons hopes to foster these developments in a genuinely peer-to-peer review process in the coming months by adding functionality allowing readers to rate and respond to the comments left by others, as well as by building links between these reviews and our social network (see Fitzpatrick 2009b), making a scholar’s reviews visible as a part of their portfolio of scholarly work. Together, we hope, these links will allow peer review to become an open, social process, one that will transform online peer review into a mechanism for collaborative post-publication filtering, helping authors to improve their work and enabling researchers to find and assess the authority of the texts they need, by working with rather than against network-based interactions.


© 2010 Centre for Computing in the Humanities

Last Updated: 30-06-2010