Rethinking Peer Review in the Age of Digital Humanities

For academics, double-blind peer review processes remain the gold standard for validating scholarly work. The value accrued by scholarship has traditionally flowed mono-directionally from peer review. In the hierarchies that govern academic hiring and tenure and promotion practices, the single-authored monograph from the distinguished scholarly press sent out for review upon completion occupies a position of prominence. Among shorter forms, the prestigious academic journal provides readily legible markers of academic quality. Yet, for scholars working in digital formats or within digital humanities, conventions governing the gatekeeping of “scholarly” work feel increasingly mismatched to the digital milieu. Therefore, digital scholarship requires consideration of the factors distinguishing it from print scholarship, along with a new approach to validating scholarship that emerges from and respects the specificities of digital work.

Rethinking peer review in the age of digital academe is a task that goes beyond the question of medium or platform to a question of epistemology. That which we call “digital scholarship” is not simply print scholarship gone digital but raises questions of genre and gives rises to its own conventions. One of the first significant interventions of digital platforms for academic work was the rise of “e-journals.” These academic journals, published electronically rather than in print, generally mirror print journals in both issue structure and article format. E-journals brought with them a host of responses, from skepticism to excitement{{1}}, accompanied by concern that recourse to the digital would decrease scholarly merit – as though there were little distinction between an academic journal online, with a review board and review process, and a blog. Yet, scholarly publishing has responded to the affordances of the digital: we have seen greater interest in open access and a boom in new journals that use digital platforms to distribute articles ahead of the publication lags that accompany print journals. Yet, many e-journals reproduce the hierarchies and values of print knowledge, relying on traditional notions of what academic working looks like.

On the other hand, “digital scholarship” is its own animal, a chimera that defies the conventions of print scholarship. Three principle differences between digital and print scholarship in the humanities require a radical revision to how we review and assess scholarly production and to how scholarly work accrues value: digital scholarship is often collaborative, digital scholarship is rarely finished, and digital scholarship is frequently “public.” Each of these qualities of digital work invites particular concerns for review, rendering digital scholarship not readily legible to tenure and promotion or hiring committees.

1) Digital scholarship is often collaborative.
Whereas creation and distribution of print knowledge in the humanities is usually a solitary task, digital scholarship is often collaborative and challenges the familiar image of the print academic in her hermitage, toiling in obscurity. The relatively greater frequency of collaboration in digital scholarship is a function of platform. Producing digital scholarship requires us to draw not only on the interpretive skills in which we were trained but a range of skills we may not possess, particularly as they pertain to the use and development of digital components of scholarship. Moreover, digital scholarship requires perhaps the most valuable of commodities for academics: time. Currently, I co-direct Digitizing Chinese Englishmen with Adeline Koh of Richard Stockton College. Koh initiated the project, which addresses colonial silences in 19th-century digital humanities scholarship by making available the Straits Chinese Magazine, a colonial English-language magazine published in Singapore from 1897-1907. Working alone, Koh found the project moving slowly. Given the nature of digital scholarship and its questionable value for tenure and promotion, those of us who work in the digital milieu often find we must steal time from tenure and promotion-worthy work to advance our projects. These challenges are even more acute for those of us who work outside of universities with very high research activity, with higher teaching loads and few institutional resources to support our research. In light of these challenges, Koh invited me to co-direct the project with her. I brought to the project not only my time and labor, but also my own skill set in TEI. Beyond project directors or principal investigators, other participants – programmers, graduate students, interns – often play critical roles in the creation of digital scholarship but traditional citation and reward structures of academe are not configured to acknowledge these contributions. Despite the efficiency of combining skills and human capital, collaboration, while expected in scientific or social science fields, raises concerns within the humanities, particularly over how to evaluate individual contributions to collaborative projects for tenure and promotion.

2) Digital scholarship is rarely finished.
The adage for traditional scholarship is that it is never finished – one simply finds an acceptable stopping point. While this is true of digital scholarship as well, digital projects may exist in phases, may be perpetually in-progress, and may never have an acceptable stopping point. The end point for digital scholarship is frequently a moving target, and there is often no single event in the process of creating digital scholarship that is comparable to the act of submitting a manuscript for review. Therefore, digital projects require new approaches to linear conventions of scholarly time. For example, completion of a phase of a digital project may be an appropriate moment for evaluation. Yet, such assessments will unlikely be the conferral of a mark of scholarly validation on the project but the source of additional information and feedback that might, in turn, be folded into the next phase of the project. Further, digital projects are difficult for search committees or tenure and promotion committees to compare to print scholarship because a CV line identifying a digital project cannot adequately communicate the amount of intellectual labor the project represents. Digital projects may require ongoing efforts for sustainability and preservation, whereas books and articles are presumed to be finished. The nature of intellectual labor itself comes into question as digital scholarship requires us to account for continued time, new versions, and preservation efforts that are less relevant to the production of traditional print scholarship.

3) Digital scholarship is frequently public.
Although more academics have taken to blogging and social media to share, promote, and collaborate on their work, scholarship in the humanities relies largely on private labor taking place behind the closed office doors. Public components of print scholarship are prescribed by academic ritual: invited talks, symposia, conference panels. The “public” nature of these acts of scholarship have been subject to question as well, with academics arguing that a conference talk is not, in fact, “public.”{{2}} Conversely, digital scholarship, particularly projects engaging web-based infrastructure, often become public early in development. This is in part because of the affordances of a web presence for digital work, the orientation of digital humanities work towards praxis, and the fact that many tools and platforms are web-based. Whereas a scholar producing a print article can delay making the work public for as long as she wishes, there is often relatively less agency for the digital scholar. Differentials of privacy and publicness for print and digital scholarship cleave to the levels of status and prestige they accrue. Print scholarship, usually not open access, lives within academic spaces, behind gatekeepers like paywalls, databases, and library archives. Its privacy is guaranteed by barriers that accrue capital for distributors. Conversely, digital scholarship may be relatively more easily available, without the mediating force of transactional capital preventing access. The monetary barriers to accessing print scholarship accord it quantifiable financial value – albeit one that provides returns for the gatekeeper rather than the author – and with financial value print scholarship accrues intellectual value. Thus the “publicness” of digital scholarship grants it relatively less value in the academic machine.

Based on these descriptors, digital scholarship threatens to displace a benign sort of academic discourse that does not trouble the value and status of print knowledge. What we must guard against in the creation and evaluation of digital scholarship, however, is reproduction of the fetishism of print that undergirds academic disciplines. We cannot assume that the standards of traditional scholarship can be easily translated for digital scholarship. Scholars in arts and media have been raising related issues of evaluation for decades. Those of us invested in digital scholarship must build on historically radical moves to continue troubling the relationship between print scholarship and academic status and value.

Digital scholarship, therefore, is best understood as part of an ongoing trend in academic discourse prevalent enough to require rethinking the production of academic value. Yet, we must be wary of fetishizing the digital as well. As the work of scholars in conversation with communities like #transformDH, Global Outlook::Digital Humanities, and Postcolonial Digital Humanities have argued, digital humanities risks reproducing gaps and silences in knowledge production around issues of difference and is not inherently free from biases. Even among traditional funding streams for digital humanities, such as the National Endowment for the Humanities, digital projects on canonical topics tend to receive greater sanction. Therefore, we need to attend to the ways that digital scholarship itself risks perpetuating more conservative elements of disciplinarity and canon as we look for ways to evaluate digital projects.

There is, in fact, growing interest in making the value of digital scholarship legible to tenure and promotion committees and search committees {{3}}. For example, we might look at tracking citations, grants, and usage statistics. These metrics demonstrate engagement with digital projects in ways that are not necessarily available for print scholarship. On the DHCommons editorial board, we are working to develop a review journal for digital projects that takes into account the specificities of digital scholarship. Anvil Academic, a partnership between CLIR and NITLE, is pioneering a platform for digital publishing that would be an indicator of peer review for digital projects. Ada itself forges new directions in peer review using both expert beta readers and community review. Journal of Digital Humanities experiments with peer review by drawing content from Digital Humanities Now before review begins. #DHTHIS provides a platform for crowdsourced evaluation of digital work. Efforts continue in many directions to define evaluation for digital work. Perhaps the most important insight when encountering digital scholarship, however, is to attend to the particulars of the digital – the possibilities for collaboration, new approaches to completion, and their public nature – viewing these not as limitations but affordances.

[[1]]Howard, J. 2010, Wed. April 7. “Scholars Increasingly Embrace Some, but Not All, Digital Media.” The Chronicle of Higher Education.[[1]]

[[2]]Kolowich, S. 2012, Tue. October 12. “The Academic Twitterazzi.” Inside Higher Ed.[[2]]

[[3]]Modern Language Association. 2012. “Guidelines for Evaluating Work in Digital Humanities and Digital Media. Modern Language Association.[[3]]

Risam, R. (2014) Rethinking Peer Review in the Age of Digital Humanities. Ada: A Journal of Gender, New Media, and Technology, No.4. doi:10.7264/N3WQ0220

This article has been openly peer reviewed at Ada Review.

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License

Roopika Risam

Roopika Risam is Assistant Professor of English and English Education at Salem State University. Her research examines postcolonial and African diaspora literatures and the role of digital humanities in mediating between them. Her work has recently appeared in Left History, Digital Humanities Quarterly, and First Monday and is forthcoming in South Asian Review and DSH: Digital Scholarship in the Humanities.

7 thoughts on “Rethinking Peer Review in the Age of Digital Humanities”

  1. This is one of the clearest and most succinct states of the union on reviewing digital scholarship. It hits most of the points that came up during the MLA2014 panel led by V. Szabo on Evaluating Digital Scholarship: Candidate Success Stories.

    I would add one more genre of evaluation and that is the NINES/ARC model which has been around now for more than a decade, if I know my history well. In order to get aggregated into that model you need to pass a thorough review. They are the still viable precursors of the ones you list above.

    One model I would like to see, (and I cannot confirm nor deny that I’m working on it) is leveraging the legitimacy and peer-review networks of existing prestigious journals to shift sub-disciplines that are beholden to them.

Leave a Reply

Your email address will not be published. Required fields are marked *

Rethinking Peer Review in the Age of Digital Humanities