Peer review plays an important role in today's scientific publishing processes, and in the public eye it is often associated to standards of quality in scientific research.
Most training material on peer review out there is very repetitive and doesn't really address fundamental questions critically. This motivated me to put together a doctoral course for early stage researchers, where I could (i) provide clear and effective practical training and (ii) stimulate a critical discussion on peer review.
Why this page
Peer review is not the same in every field, everybody knows this. But how? After beinging on virtually every course/webinar/publication I could find online, I realised that nobody adopts a comparative approach to peer review. So I reached out to some of my esteemed colleagues and ask them if they would share a summary of their approach. This page is the first step towards a larger project that addresses peer review historically, technically, ethically, and from every discipline's perspective.
What you see on this page
I interviewed twelve colleagues from different disciplines and different countries. I do not cover every discpline, but there is a fair representation of "distant" disciplines like linguistics and chemistry, material engineering and philosophy.
I asked the same questions to each of them:
- What is your field?
- As a reviewer, how do you approach peer review in your field?
No follow-up questions. The full transcripts are available at the bottom of this page.
Share and speak up
Please share with the hashtag #DissectingPeerReview. You can send your comments to my email. I am Federica Bressan and this is my research profile.
Luca Viganò
King's college, London, UK
Peer review boils down to a question of novelty, rigour, impact.
George Dunn
Indiana University-Purdue University, US
Institute for Globalizing Civilization at Zhejiang University, China
Summarizing the article in your own words is a good way to see if the question, arguments, and conclusions are clear. Besides the work's shortcomings, I always point out its strengths.
Enclosing relevant authoritative sources in the bibliography is very important. As for the methods, the reviewer may assume the author's goodwill and trust their description.
[Q] What's your field?
My field is Digital Humanities, specifically network analysis.
[Q] As a reviewer, how do you approach peer review in your field?
When I look at a manuscript that has been sent to me for review I consider a number of things. Most importantly I think about, as a reviewer, the overall argument, and by argument and not only mean the logic of the argument, but does the specific research question and the theoretical framework that they suggest to explore this research question connect also with the methods and the data that's been collected. And a lot of times I encounter situations where this is not the case. Where the research question is very clear, there's definitely a gap to be filled, there's a clear theoretical explanation as why there's this gap, should they be looking to fill a theoretical gap or an empirical gap, and how we don't understand something in practice, but then something, then the method and the methodological approach doesn't fit how we should measure or approach or accurately operationalize these concepts or even that the data itself, that there are questions about the data collection or the validity or the reliability of the data that was collected. Does the combination of these elements need to be sufficient. Of course, there will be stronger elements of the piece overall, that perhaps the gap is very clear, but the data and the collection and the conclusions are sufficient enough to warrant such an exploration? Or the other way around, so that I see that the data is an excellent data set, the method is very clear, but that actually you're contributing to something else. So I try and seek to help the authors refine that argument whether it's on one of those four aspects of the clear research question, the theoretical framework, the methods, data collection.
Brian Booth
Antwerp University, Belgium
[Q] What's your field?
So, I'm a computing science researcher and I focus specifically on medical image analysis.
[Q] As a reviewer, how do you approach peer review in your field?
So, in my field you're looking for a few things in peer review. I think the most important is the technical correctness of the work, making sure that everything is presented properly in terms of the equations, in terms of the algorithms. Beyond that, the validation has to be up to a level that is convincing. So making sure that you validate on enough data and making sure that you've tested enough corner cases. And then on top of that there's a need to show novelty and impact as well. So, that's really the part where you run into the biggest challenges with the peer review, because it's not often that, you know, researchers in our field really motivate that their work is novel and really sort of make it clear what the contributions are. So, that's something that I look for in peer review and I sort of encourage people to write with that in mind.
Ana Paula da Costa Ribeiro
Instituto Superior Técnico, Lisbon, Portugal
[Q] What's your field?
My field is chemistry and nanotechnology, in particular catalysis, chemical engineering, green chemistry, and the production of nanomaterials.
[Q] How do you approach peer review in your field?
Basically, besides the normal checking of the values and the accuracy and if the experimental part, if I'm talking about original work, is well described, so that anyone in any place in the world can do it. Because that is important in science, that another person that has nothing to do with the original work is able to check it. So that is one of the most credible ways of doing a validation of a process in science. The other part that is important is really if the message that the authors want to send is well explained, because usually when you do an article or a review, you're talking about some topics that require improvement and your contribution to that issue is done by the article. So, if you're not able to say it properly, you are not able then to express it, and probably the article lacks the aim and that makes it usually not so easy to understand. If it's not easy to understand, it may be work that will not have the same validity.
Luca Viganò
King's College, London, UK
[Q] What's your field?
I'm a computer scientist, but I work mainly in the area of cybersecurity. In particular, on how to apply formal methods and artificial intelligence techniques to cybersecurity problems.
[Q] How do you approach peer review in your field?
Typically peer reviewing works by trying to answer three questions: is the work original? And of course there are different ways in which one can answer that question. It could be original because it is a fundamentally new idea or it is original because it builds upon previous work, maybe by the authors, maybe by other people but extends it in a significant way. That is the first characteristic. If your work is not original, in one of these forms, then it will be very difficult to have it accepted by your peers. The second question is rigour: to what extent has your research been carried out in a rigorous way, to what extent is your paper presenting your research in a rigorous way. Now again, there are many different ways in which you can answer that question. For people like me, working more on the mathematical aspects of things, of security in particular, then the question would rather be: what kind of definitions, what kind of formal results do you have? Are you able to prove some theorems? Are you able to prove some properties about the security or the insecurity of the system that you're considering. But there are also other forms of rigour: for instance, and I've happened to the experience that as well, if your paper is a bit more applied, then people will expect rigour through experiments, through tests carried out maybe with users may be without users, but systematic tests assessing the security of the systems in such a way that the peers can reproduce these tests. Much in the same way that they can try to reproduce the proofs that you have done in the more mathematical side. The third question is significance. This basically means impact, to what extent is your work interesting. What are the consequences of your work? What is going to be the impact of the work? This could be just academic impact, it doesn't have necessarily to be practical impact for society or for industries. You know, it could be that you are advancing the state-of-the-art so that other academics can build upon it or it can actually have a very practical resonance. So, for instance, you know, to really make secure existing systems that normal people use or it could even be impact in the sense that you influence governments, you influence society, policymakers and ultimately are able to reach the users, the laypersons who are actually going to be benefiting from your work.
Thorsten Ries
Ghent University, Belgium
[Q] What's your field?
I'm a literary scholar, I'm working in German literature and Digital Humanities, digital forensics, born-digital archives.
[Q] As a reviewer, how do you approach peer review in your field?
It's highly contextual and it's often governed by questions of practicalities, of editors who are desperately looking for someone who is able and even willing to do a peer review on specific paper, on a specific topic. Most of the more classical humanities articles and contributions don't come with a dataset. They come with references to authoritative editions and so on and so forth. More the Digital Humanities minded contributions might, but not always come with datasets that you could have a look at. if I'm perfectly honest, I did not go all the way to reproduce the results so far, if I had the chance to do so. I peer reviewed on the goodwill assumption that the analysis or that the method has been applied as described, and I would have focused on the described method and the analysis of results, as a peer reviewer.
Alejandro Higueras
National Distant Education University, Madrid, Spain
[Q] What's your field?
My field is psychology and I specialised in addiction neuroscience.
[Q] As a reviewer, how do you approach peer review in your field?
So, normally addiction neuroscience is an experimental science so we are really concerned about the solidity of the methods. The methods have to be sound and solid and reproducible. Reproducibility is a really big issue in our field. And then, normally people are very concerned about the conclusions of the papers. They have to derive from the data and people tend to be scared about conclusions being very speculative. So that's a major issue also during peer review. A very important point for us is the statistical analysis, so that tends to be a really important issue, which actually raises important points about papers, and sometimes I recall being redoing stats for almost a year, thus receiving a very important and useful recommendation from reviewers. And when I do the reviews myself, I tend to focus on statistics quite a lot as well. And also my field deals with biochemical analyses, and we also tend to focus a lot on the graphical representation, because there have been some issues with forgery. And now we all really really paying attention to that, and really being really careful with these issues.
George Dunn
Indiana University-Purdue University, US
Institute for Globalizing Civilization at Zhejiang University, China
[Q] What's your field?
Philosophy with a concentration on ethics, philosophy of religion and philosophy of popular culture.
[Q] As a reviewer, how do you approach peer review in your field?
When preparing a peer review, I generally start by trying to summarize the article in a few sentences without editorializing about it in any way. My aim is to identify the questions that addresses and briefly to summarize his argument and inclusions. When evaluating the article I look at whether those questions are interesting and important and whether its conclusions in addition to being original follow rigorously from the author's arguments, other objections that the author hasn't anticipated; are all the steps of the argument clearly presented, and all of the assumption spelled out? I'm mainly concerned with how well the argument holds together logically. In addition, when evaluating the article's conclusions, I consider whether they're likely to be of interest to other researchers in the field. So does this article make a genuinely original contribution? Does it open up possibilities of new directions in research that could prove fruitful? Other considerations are whether the quality of the writing is up to professional standards, whether more background information is needed and whether any other relevant considerations are missing. When writing the review why I always try to say what I like about the article even if in the end I'm not going to recommend it for publication. Also when giving critical feedback and I try to be as specific as possible. And couch my criticisms as recommendations for improvement. Sometimes, however, an article maybe just so carelessly written and its argument so shoddy and irremediably bad that all I can do is point out its failings.
Acatia Finbow
Independent researcher, London, UK
[Q] What's your field?
My field is cultural heritage, specifically museology, and in particular the documentation of art works in time based media.
[Q] As a reviewer, how do you approach peer review in your field?
So, because my field doesn't necessarily have data as such we're often looking at a particular artwork, it can often be quite interpretive which means that when we get peer review, we don't always have someone who agrees with our perspective on things. This means that quite often the suggestions we're getting are things like suggested additional reading, whether we can consider somebody else's perspective on the artwork or on that particular time period. And quite often when you're responding to that either you do that additional reading, it gives you a new point of view on the artwork that you're looking at, or equally, by responding to peer review can be asserted why you don't necessarily agree with that point of view. A lot of the work we do in response to the review can be acknowledging the difference that there is in the field and incorporating them into our own perspective. You don't necessarily have to agree, but it's important to acknowledge when there are those different points of view and assert that within what you're writing.
Gareth O'Neill
Leiden University, The Netherlands
[Q] What's your field?
Ok, so, linguistics and in particular theoretical linguistics. I work on Celtic languages.
[Q] As a reviewer, how do you approach peer review in your field?
So, I usually, I mean I take it twofold, I guess. I usually look at the article critically to see if they have followed their own process, if they achieved what they set out to do, and the methodology and if there's data, you know, to the extent that I have access to the data, check the data, does it make sense, does it actually produce these results? And I tend to try and constructively give comments on this article. So, I don't see it necessarily as a kind of antagonistic approach but rather, reading this, I'm trying to understand it, make sure it's methodologically sound, and then I try to let's say help the author. So, if I see that maybe they could have done this, or maybe they miss some literature and so forth, then I will suggest this. I guess that differs per author.
Murat Gunes
Independent researcher, Paris, France
[Q] What's your field?
Physics, material science, in particular thermoelectricity.
[Q] As a reviewer, how do you approach peer review in your field?
We have different articles received related to materials development or thermoelectric measurement system development for measurements of the thermoelectric properties and also the modules that are actually used in practical applications. So, for example, for thermoelectric materials, what you are looking for actually whether the material is actually synthesized in a way that is required, and the data, when they get the results from the measurements, whether it matches with the literature, or comparable, and also whether they support with the data the idea that they claim in the article. But different journals have different scopes, that's why also the article, the obtained results should also be in the scope of the journal. Could be kind of repeating experiments, systematic studies, or can be a breakthrough research that should be in another journal. So these are the things that you are looking for.
[Q] What's your field?
Hi, my field is chemistry, and specifically I work in organometallic chemistry and catalysis.
[Q] As a reviewer, how do you approach peer review in your field?
Well, when you review a paper in chemistry, one of the things that you have to check or what you want to see is that most of the time new chemical compounds are being reported. So, what you want to do as a reviewer is to make sure that the assignment of these compounds is done correctly, that the experimental procedures makes sense and that they are reproducible. For example, when you read a paper in chemistry, normally you have some preps that show how they made the reactions, and you need to make sure that all the conditions are there so that if someone in the future wants to do the same compound, they can easily follow the recipe that is published. Another thing that is important in my field is to make sure, for example, that if you report reaction rates or if you report reaction mechanisms, that they make sense. What I want to say is for example that normally when a reaction happens, you go from A to B, is not usually a one-step, normally there are several steps involved, and sometimes some chemists like to understand how reactions happen, and they do a series of experiments to make sure that when you go from A to B, you go through all of these intermediate, and then they draw a picture of how this happens and then as a reviewer you want to make sure that, well, the experiments that were done, they correspond to the conclusions that the person is drawing on the paper, but I think that's also general in all fields.