Writing a Review The Big 9 to Answer

1. Read the introduction/background/results/conclusion and summarise what you think the

paper is about based on those sections in one paragraph.
important because? you need to get a basic idea of the paper, and you need to be able to
validate that what is being claimed is also what will be presented/solved in the paper.
is the research idea original/novel?

is this idea
appropriate for
this conference?

remember: authors should be clear about this!

what is/are the question(s) being

addressed by the author(s)?

is the current
state-of-the-art
well-represented?

2. Validate the claims in the body of the paper. Are the stated research questions addressed?
Are the authors advancing on the state-of-the-art? hint: best to do this the next day.
important because? the claims need to match up with the actual contents. Do wait a day
before you try to tackle this question. Most papers are evolutionary and come up with
small improvements. At first, you might be underwhelmed by the small contribution and at
the same time feel intimidated as you think you are not an expert. Both are likely not true.
3. Once you have validated that the claim matches the contents, you are ready to judge the
novelty of the paper. Give your score (see below), and then motivate your score in a few
sentences. Use facts to motivate your score, not your emotions!
(7) The paper is highly original, proposing an entirely novel view, idea or problem.
(6) The paper has several novel and surprising contributions.
(5) One idea surprised me with its originality, otherwise solid contributions.
(4) The paper offers worthy contributions, none of which are surprising. (i.e. next step)
(3) The paper only offers minor variations to existing ideas.
(2) The paper does not advance our current state of knowledge.
(1) This work is behind on the state-of-the-art, or is a duplicate of existing work.
important because? the novelty of a paper is the single most important criteria to determine if
a paper should be accepted. Dont expect (all/any?) papers to be revolutionary though!
4. Validate and motivate the technical quality of the paper. You have to check if the results are
technically sound and detailed enough. Are there any obvious flaws? Are the claims wellsupported by either their own novel contributions or related work? Are the results
reproducible? Did the authors discuss both strengths and weaknesses of their approach?
important because? good work is firmly rooted in sound theory and methods. Small ideas that
are well-supported are more useful than big ideas that are subsequently proven wrong.
is the approach
theoretically
sound?

can you think of a better way to

address the papers questions?
are relevant papers referenced?

are the
experiments
appropriate?

5. Score and comment on the significance of the paper. Take the novelty and the technical
quality into account when doing so, and try to estimate the impact of this work.
(7) A high-impact, far reaching (to other domains) contribution.
(6) Very significant contribution that advances the state-of-the-art.
(5) A solid contribution to a relevant problem.
(4) A reasonable contribution to a minor problem.
(3) Unsure how this paper is relevant.
(2) Not really a relevant paper for this conference/journal.
(1) Out of scope.
important because? we publish our work for others to read, not for it to disappear in oblivion.
6. Give your feedback on the quality of writing. While syntax and grammar plays an important
role, this is also about the structure of the paper, the good use of figures/tables, and the clarity
of the paper overall. Always motivate your decision, and give a reasonable amount of input.
(4) The quality of writing is excellent; few typos, good structure, clever use of extras.
(3) The quality of writing is good, but could use some editing.
(2) The quality of writing is only marginal to ok, it needs a fair amount of editing.
(1) Quality of writing is poor and considerable editing is needed; a.k.a. Google translation
important because? good ideas buried in an unreadable paper are not very useful at all.
7. Motivate and give your score for the overall quality of this paper. To do this, take all of the
previous questions into account, but it need not be a simple weighted average.
(7) An excellent paper, a very strong accept. I will fight to get this paper accepted.
(6) A very good paper that should be accepted. I vote for and argue for its acceptance.
(5) A good paper overall that should be accepted. I vote for acceptance.
(4) A decent paper overall. Accept or reject, either way is fine by me.
(3) A weak paper that is just not good enough. Reject, yet could be persuaded otherwise.
(2) A clear rejection. I vote for and argue for rejecting it.
(1) Reject outright. I will fight to get this paper rejected.
important because? this single score makes it easy to compare different reviews.
8. What is your own confidence in your ability to review this work?
(3) Expert; I work on the same topic or have worked in it in the recent past.
(2) Knowledgeable; this topic was a smaller part of one of my previous works.
(1) Educated guess; I saw something about this during my BSc/MSc studies. No?
important because? knowing your level of expertise helps the Program Committee (PC)
members when there are conflicting opinions. Do not be timid about your expertise level,
or the PC member may ignore or overlook your opinion.
9. [optional] Provide comments to the Program Committee (PC) members. This is particularly
relevant when you suspect (self-) plagiarism, when you have concerns about your own bias, or
when you want to rank the papers you have been reviewing but the numbers at your disposal
for the scores are inadequate/do not capture the full picture.
At all times motivate your scores and scrutinise the work, not the authors. Your comments should
hold up to the same scientific rigours and should be refutable. Always write in a way that gives the
authors the chance to discuss their point of view (you may very well be wrong!). Why not give some
suggestions to the authors on how they can improve their paper?