Quality assessment tool for systematic review of qualitative

Appendix to Krlner et al. Determinants of fruit and vegetable consumption among children and ado
Int J Behav Nutr Phys Act 2011. (See also table 2 in the paper)
Quality indicator:
Aims
Aims and research questions are explicitly stated
Qualitative approach appropriate to answer
research questions
Preconceptions
Explicit theoretical framework or literature review
and/or pre-study beliefs
Information on how theory is used (NA if no
theoretical framework)
Sampling procedure
Explicit sampling strategy of field sites and/or of
children

Recruitment strategy: how?

Recruitment strategy: by whom?
Explicit justification of sampling strategy
Sampling strategy reflects the study purpose
Sample size provided or can be estimated

Non-participation described/response rate (NA if

voluntary sample)
Sampling/data collection continued until point of
data saturation
Ethical concerns: explicit statement
about
Informed consent (parental or child)
Anonymity and confidentiality
Ethical approval/review
Sample characteristics: Explicit and
sufficient description of
Gender of child participants

Age of child participants or school year

Socioeconomic background of child participants

Ethnic background of child participants

Other study-specific characteristics of child

participants

Data collection
Data collection method (e.g. focus groups,
observations) stated
Explicit rationale for data collection method
Data collection methods adequate to answer
research questions
No. of focus groups, interviews, observations
provided
Size of focus groups described or average can
be estimated
Composition of child focus groups/interviews
described
Explicit rationale for focus group/interview
composition
Interview setting described
Interviewer described (who?, how many?)
Duration of interviews, focus groups,
observations described
Interview guide
Interview guide used
If yes: Partly described (key questions)? Y, fully
described? YY
Analysis
Reliability/consistency)
Explicit information of audiotaping of interviews
Explicit information of transcription of interviews
Y, verbatim: YY
Communicative validity
Analyst described (who?, how many?)
Clear description of analytical method?
Explicit analytical approach (data-based or
theory-based)
Analytical procedures appropriate to the
research questions
Explicit rationale for choice of analytical
procedures
Sampling strategy/ child focus group
composition is used in analysis
Findings/ presentation of findings
Clear presentation of findings
Authors voices can always be distinguished
from informants voices
Sufficient inclusion of quotes to support findings
Clear description of selection and edition of
quotes
Different child participants' views can be
distinguished
The stated conclusion is supported by the
findings

Relevance: Findings/conclusions illuminate the

research questions

Internal validity
Description of validity and pilottesting of applied
instruments/guides

Explicit strategies for validating presented

findings
Researcher/analyst triangulation

Method triangulation

Source triangulation

Theory triangulation

Peer debriefing/audit trail

Member checks/respondent validation

Attention to negative or deviant cases

External validity
Discussion of transferability (applicability of
findings in other contexts)
Explicit reflections on selection bias/nonresponse of children
Discussion
Adequate attention to previous knowledge and
what the study adds
Findings provide new insight on potential
determinants of fruit and vegetables
Discussion of limitations of qualitative study
Pragmatic validity
Discussion of implications for research and
practice
Quantitative quality score: Total no. of quality
requirements met
Qualitative quality score evaluated by
reviewers (H: high, M: medium, L: low)

Abbreviations: Y indicates 'yes, information is provided', NA indicates that the criterion is no

number, SEP= socioeconomic position.
The quality assessment tool is inspired by the following papers:

Fade SA: Communicating and judging the quality of qualitative research: the need for a new l

Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, Oakley A: Applying systematic revi
public health research. J Epidemiol Community Health 2004, 58: 794-800

Malterud K: Qualitative research: standards, challenges, and guidelines. Lancet 2001, 358: 483

nt tool for systematic review of qualitative studies

rminants of fruit and vegetable consumption among children and adolescents: a review of the literature. Part II: qualitative studies.
. (See also table 2 in the paper)
Description:
Are aims and research questions explicitly stated in the manuscript?
Reviewer judgement: Are qualitative methods appropriate to answer the research questions (or do quantitative
methods seem more appropriate?)
Are the theoretical perspectives/ideas, empirical background and pre-study beliefs/hypotheses used for data
analysis and interpretation made explicit?
Do the authors explain how they use their theoretical framework? e.g. to develop interview guide, to categorise
results, to discuss results
Is the sampling strategy clearly stated? e.g. school-, family- or community-based? theoretical sampling?,
chain/snowball sampling?, convenience/ pragmatic/volunteer sampling?, systematic/criteria-based sampling?,
maximum variation sampling?, homogenous sampling?, extreme or deviant case sampling?

Is information provided on how participants were recruited e.g. through schools, households, quantitative
surveys, newspapers?
Is information provided on whom recruited the participants e.g. project staff, school teachers?
Do the authors provide the rationale for their choice of sampling strategy?
Does the sampling strategy reflects the study purpose?
Is information on number of participating units (e.g. schools) and number of students provided or can it be
estimated from information on number of focus groups and number of students per focus groups?
Is non-participation described : How many schools/students were invited to participate and how many declined
to participate? Are reasons for non-participation provided? NA if volunteer sampling is used.
Was the sampling and data collection continued until theoretical saturation /informational redundancy was
reached i.e. new sources of data did not reveal anything new about the analytical categories?

Was informed content sought and granted from children and/or parents?
Were participants anonymity and confidentiality ensured?
Was approval from an appropriate ethics committee received?

Was information provided on the gender of participants? It is sufficient to know whether the student sample is
gender-heterogeneous or homogeneous. Exact rates are not necessary.
Is information provided on the age of participants? It is sufficient to know whether the student sample is ageheterogeneous or homogeneous. Exact rates are not necessary.
Is information provided on the SEP of child participants? It is sufficient to know whether the student sample is
SEP-heterogeneous or homogeneous. Exact rates are not necessary.
Is information provided on the ethnic background of child participants: It is sufficient to know whether the student
sample is ethnic-heterogeneous or homogeneous. Exact rates are not necessary.
If the sample possesses certain study specific-characteristics, have they been reported? E.g. diabetic children,
children in and outside a youth gardening program

Are the methods for data collection described? e.g. focus group, face-to-face interviews, ethnographic
observations /observational fieldwork
Do the authors provide the rationale for their choice of methods for data collection
Reviewer judgement: Are the selected data collection methods adequate to answer the research questions?
Are the numbers of focus groups, interviews, and/or ethnographic observation sessions provided?
Is the size of the focus groups described or can the average size be estimated from number of focus groups and
sample size?
Is the composition of focus groups/interviews described in detail? e.g. are the groups mixed by gender?, age?
Are the face-to-face interview conducted individually?
Do the authors provide the rationale for their choice of focus group composition? e.g. why it is genderhomogeneous or age-heterogeneous
Is the setting where the interviews/ focus groups are carried out described? e.g. at school, in classroom, in
students home
Is information provided on who and how many conducted the interviews? NB this information might be found in
the author contribution section in paper.
Is the duration of the interviews/focus groups/observations described?

Has an interview/discussion guide/questioning route/ checklist been used in the study?

Is the interview guide fully described in the paper e.g. presented in table (2 points, YY), or partly described e.g.
only key questions/key topics (1 point, Y).

Are the discussions/interviews audio-taped? (versus being based on interviewers memories/notes)

Are the discussions/interviews transcribed (1 point, Y) or transcribed verbatim (2 points, YY) so the evidence can
be inspected independently and analyzed by others? (versus being based on interviewers memories/notes)

Is information provided on who did the analysis and how many?

Are the methods used to analyse data clearly described? e.g. grounded theory, constant comparative methods,
phenomenological analysis, content analysis
Is information provided on whether the themes and concepts are derived from the data (data-based analysis) or
from the theoretical framework/interview (theory-based analysis)?
Reviewer judgement: Are the analytical procedures related to/appropriate to the research questions?
Do the authors provide the rationale for their choice of analytical procedures?
Do the authors use the sampling strategy / focus group composition in the analysis? E.g. to examine age-,
gender-, SEP or ethnic differences?

Has a clear presentation of findings been provided?

Can we distinguish authors voices from informants voices?

Are the findings supported and enriched by adequate use of quotes/original data?
Is information provided on how the authors have selected quotes and how the quotes have been edited?
Are readers able to differ between different informants' information? e.g. between findings based on students
reports versus on parents or school staffs' report?
Do the findings support the stated conclusion?

Are the findings relevant with respect to the aim of the study? Does the conclusion answer the research
questions?

Is information provided on the validity of the applied instruments/guides? Have the authors ensured that the
questions are meaningful to the age group e.g. through pilot studies, by involving experts and/or by developing
and adjusting the instrument concurrently during the study?
Have the authors applied any of the strategies below for validating presented findings?
Have the coding strategies, findings and interpretation of data been cross-checked by two or more independent
researchers, at team meetings, or at supervision sessions?
Have the research question been examined from different angles to give a comprehensive picture of the
phenomenon of interest by combining different data collection methods e.g. focus groups and field
observations?
Have the research question been examined from different angles to give a comprehensive picture of the
phenomenon of interest by combining different data sources e.g. parents, staff and students?
Have the research question been examined from different angles to give a comprehensive picture of the
phenomenon of interest by applying different theoretical perspectives during analysis?
Have the researchers exposed their rationale and methodological considerations to an experienced protagonist
playing the devils advocate. Can a peer understand the rationale of analysis?
Have data, concepts/constructs and interpretation been checked with respondents?
Did the investigator give evidence of seeking out observations that might have contradicted or modified the
analysis and the hypotheses? Have the researcher reported the negative cases and tried to explain why the
data vary?
Do the researchers discuss if the study findings are likely to be transferable to other settings or with other
subjects?
Do the authors discuss selection bias in relation to the interpretation of data?

Reviewer judgement: Is adequate account taken of previous knowledge and what the study adds?

Reviewer judgement: Do the findings provide new insights?

Do the authors provide a balanced discussion of practical, methodological, and theoretical limitations and
strengths of the study?
Can the results be used by other?
Summation of numbers of Y. NA is also counted as a yes to do justice to the studies. Lack of information is not
viewed as a methodological limitation if the criterion is not applicable in a study.
The reviewer's subjective judgement of the scientific quality of the study based on the review of methodological
issues and the reviewer's scholarly experience -similar to a peer-review/referee process.

es, information is provided', NA indicates that the criterion is not applicable or relevant for the study. NB= notabene, No.=
c position.

s inspired by the following papers:

d judging the quality of qualitative research: the need for a new language. J Hum Nutr Diet 2003, 16: 139-149

ees R, Shepherd J, Brunton G, Oakley A: Applying systematic review methods to studies of people's views: an example from
demiol Community Health 2004, 58: 794-800

ch: standards, challenges, and guidelines. Lancet 2001, 358: 483-488.