DHThis: An Experiment in Crowdsourcing Review in the Digital Humanities

In 2013, Martin Eve, Jesse Stommel, Roopika Risam, Alex Gil and I{{1}} launched DHThis.org, an experiment in crowdsourcing the “best” work in the digital humanities{{2}}. DHThis is the first entirely crowdsourced outlet for digital humanities (DH) news. There are no comparable models to DHThis. All existing sites which aggregate DH content still run on an editorial model. DHThis flips this model, shifting control of new developments in DH to wider publics. Using a Slashdot-style system of user engagement, DHThis gives registered users the opportunity to upvote and downvote articles and give karma points that reward active (and useful) participation in the community. We built DHThis on an ethos of open access and open engagement, and to provide an ongoing forum for defining DH in the moment.

DHThis originated in a set of conversations among the team about how knowledge is produced, distributed, and consumed within DH.  The “digital humanities” has come under fire in the last five years for its exclusivity{{3}}, despite its attempts to redefine itself under the metaphor of a “big tent” which able to encompass much broader forms of work. This exclusivity results in the general predominance of certain types of digital humanities work (big data analysis, topic modeling, digitization) over others (e.g. cultural-studies nuances forms of digital humanities criticism). Our team decided to create DHThis to answer the research question: is the digital humanities community growing to the extent that it is much more diverse than it used to be? If so, can crowdsourced curation serve as a means to illustrate the diversity present within the “big tent”? We created DHThis to attempt to answer these questions.

We launched DHThis on September 13 2013, using a basic installation of the Pligg CMS. People were invited to register for free and to submit content. Any registered user can vote on the content using a thumbs-up or thumbs-down function. To appear on the front page of DHThis, a submission needs to have 5 thumbs-up votes. Votes are also accorded by “karma”—which users can accrue through actions including submission, up voting and comments—all aspects of functioning well within a community{{4}}. Users need a minimum of 100 karma points before a post they submit will appear on the front page. Once users register, they are allowed to submit links and add their own descriptions as to why their posts should be upvoted.

The “New” page is where posts first appear after they have been submitted:

Image 1

If a post gets 5 “thumbs up” from other users (also known as upvotes), it will then be published on the Home page of the system. Posts that do not accrue 5 or more upvotes in ninety days will not be published on the home page.

Image 2 

DHThis allows users to view which users have upvoted or downvoted particular stories, and to leave comments about the post. This becomes visible after a user clicks on a link on #DHThis. Clicking on the “Who Upvoted” tab will reveal a screen similar to the one below, which lists the users who have up voted the story. There will also be a “Who Downvoted” tab if the story has received downvotes.

Image 3

To foster the creation of communities and subcommunities, DHThis also allows users to create their own groups. By creating a group, users can share specific content to a self-selected group of people:

Image 4

When the project was first launched, the following criticisms and their counter-arguments were made {{6}}:

Critique 1: Upvoting and downvoting is overly simplistic, and an insufficiently complex metric for the complexities of academic work.

Thoughts: What is the end result of peer review other than a thumbs-up or a thumbs-down? When a peer reviewer, editor or editorial board judges whether your article or book deserves to be published, these decisions also arguably boil down to thumbs-ups or thumbs-downs. One might argue that these decisions come at least with comments for revision, but there is also a comments function within the DHThis site that allows users to exchange commentary on the various links which they have upvoted or downvoted.

This is however only one perspective of peer review. There are indeed generous reviewers who view the Revise & Resubmit procedure as a way to really improve the work, rather than issuing it an ultimate thumbs-down. We will continue to think about how we can take this into account while considering the project’s subsequent iterations.

Critique 2: Is DHThis really just a popularity contest?

Thoughts: Traditional peer-reviewed academic work, unfortunately, is not immune to the popularity contest. Academic publishing is tied with the larger economic market. However, an interesting point to note is that one’s academic reputation or popularity do not result in an aggrandizement of user power in terms of upvotes or downvotes, as they would in a traditional review circle. In this sense the power structures of voting in DHThis offer an alternative to traditional peer review.

However, it would be useful to observe whether the content that is highlighted on DHThis actually diverges from what has been showcased in more traditional outlets. This information will only become viable, however, when the DHThis community reaches sufficient critical mass and activity.

Critique 3: DHThis claims to be original, but the team that started DHNow also began with a crowdsourcing model, and only switched back to an editorial model upon realizing that the selected content was not diverse enough.

Thoughts: We are grateful for the historical precedent that DHNow has set. However, we launched DHThis to test if the digital humanities community has grown to the point where a crowdsourced platform will actually reflect the diversity of this community—diversity in terms of disciplines, methods, bodies and representations. Arguably, during the period where DHNow was first launched, the community was less diverse than it is today. However, with the growing popularity of the digital humanities field, and the flowering of multiple digital humanities, the tastes of the community may have changed. We are thus using DHThis as an experiment to see if this hypothesis—that the DH community has grown more diverse—holds true.

Critique 4: The downvoting function of DHThis is not encouraging to junior scholars, who might feel more embarrassed about their work being critiqued in public. Also, shouldn’t downvoting be anonymous to encourage honesty?

Thoughts: Junior scholars will have to ultimately deal with “downvoting” in the traditional research arena, which means either rejections or revise-and-resubmits. Downvoting is “anonymous” in the sense that one can sign up for a user profile with whatever information one chooses. However, whatever votes or comments a user makes will be tied to their user profile. We employ this adjusted form of anonymity to help mitigate some of the venom that can come from the anonymity within the traditional peer review process.

Critique 5:  DHThis is supposed to be a place which encourages community building. If so, why does it borrow from the Slashdot and Reddit platforms for a model, given their infamous reputations for trolling and internet flaming?

Thoughts: Our intentions with DHThis were to experiment with crowdsourcing in the digital humanities. The entire idea is focused around the idea of a crowd rather than a small elite group—and Slashdot and Reddit are some of the existing few models out there for this. As our platform grows we might have to implement some kind of moderation system to work on the possibilities of trolling and flaming. Currently,  however we do not have enough users for this to be an issue (our biggest problem currently is with spam).

In Conclusion

As of January 26 2014, DHThis has currently 208 users. These users have submitted a total of 148 submissions, 45 of which have been published to the front page. Reports has shown that activity is generally limited to a small group of users, who are most actively posting and commenting, particularly during the project’s launch in September 2013, and any time the project is demonstrated. We hope that this small group will increase as the project grows.

Given these statistics, the DHThis community has not achieved sufficient mass to decide whether the actual research question of its creators—whether the digital humanities has grown and is diverse enough as a community to make content curation through crowdsourcing a viable method for determining value in the field.

We intend to continue on with the current iteration of DHThis for a calendar year, and try and encourage more people to join and participate in the community. The experiment might never be able to show sufficient results if we do not generate enough user mass [7]. But if it does, what happens will be able to tell us a great deal about the changing nature of fields, and the different structures of gatekeeping.

[[1]] #DHThis is a collaborative project by the five of us, but Adeline Koh originated and spearheaded the project.[[1]]
[[2]] We define “best” work here as the most notable work that the users of #DHThis have curated. Existing models define “best” work through individual journal editors or traditional blind peer review. This system of curation employs a form of review as well, but one which is not blind on either end. We understand though that this is a conflicted term to use. We hesitate to use the term “popular” instead because it implies that group curation is somehow less effective regarding in sourcing valuable content than individual editorial curation, but are open to different terms. We thank Carol Stabile and Bryce Peake for this nudge to clarify in the review process.[[2]]
[[3]] The most comparable outlet to DHThis is Digital Humanities Now, or DHNow (http://digitalhumanitiesnow.org/), published by the Center for History and New Media at George Mason University. DHNow aggregates content from digital humanities feeds, but highlighted content (the “Editor’s Choice” content that appears on main site) are chosen by an editorial group.[[3]]
[[4]] The “karma” module is part of the Pligg CMS and can be edited to suit the needs of different websites. Find out more about karma here. Currently #DHThis uses the default settings for karma, whereby submitting a story that becomes published gains 50 karma points, submitting a story 15 points, submitting a comment 10 points, voting 1 point. If a story is discarded users will be deducted 250 karma points, 50 points deducted if their comment is removed, and 10 000 deducted if their story or comment is marked as spam.[[4]]
[[5]] See for example: Alexis Lothian and Amanda Phillips, “Can the Digital Humanities Mean Transformative Critique?” Journal of E-Media Studies, 3.1 (2013): n. page. Web. DOI:10.1349/PS1.1938-6060.A.425 <http://journals.dartmouth.edu/cgi-bin/WebObjects/Journals.woa/1/xmlpage/4/article/425>; Adeline Koh and Roopika Risam, “Open Thread: The Digital Humanities as a Historical ‘Refuge’ from Race/Class/Gender/Sexuality/Disability?” Postcolonial Digital Humanities, 10 May 2013, Web. <http://dhpoco.org/blog/2013/05/10/open-thread-the-digital-humanities-as-a-historical-refuge-from-raceclassgendersexualitydisability/>[[5]]
[[6]] Some of these critiques and counterarguments took place on Twitter, but most were expressed to the team orally and were not published in any way.[[6]]
[[7]] The author would loosely define this results and mass as accruing at least 1000 non-spam users with a total of 1000 published posts over a period of 2 years, but is open to different interpretations as well.[[7]]

Koh, A. (2014) DHThis: An Experiment in Crowdsourcing Review in the Digital Humanities. Ada: A Journal of Gender, New Media, and Technology, No.4. doi:10.7264/N3RX99C5

This article has been openly peer reviewed at Ada Review.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License.

Adeline Koh

Adeline Koh is Director of DH@Stockton and an assistant professor of literature at Richard Stockton College. Her work spans the intersections between postcolonial studies and the digital humanities, 19th/20th Century British and Anglophone Literature, and Southeast Asian and African studies. She runs the Postcolonial Digital Humanities (#DHPoco) website with Roopika Risam and writes for the ProfHacker column at the Chronicle of Higher Education. She has published on race, revolutionary cinema, British and Anglophone literature, Southeast Asian feminism and the digital humanities.

2 thoughts on “DHThis: An Experiment in Crowdsourcing Review in the Digital Humanities”

Leave a Reply

Your email address will not be published. Required fields are marked *

DHThis: An Experiment in Crowdsourcing Review in the Digital Humanities