- (2011) Volume 8, Issue 3
Sarah Salway BA(Hons) MSc PhD1* and Ruth Barley BTh(Hons) MRes2 and Peter Allmark BA(Hons) MA PhD3 and Kate Gerrish PhD MSc BNurs4 and Gina Higginbottom BA (Hons) MA Postgrad Dip (Ed Studies) PhD5 and Mark RD Johnson MA PhD DipHE6 and George TH Ellison BSc(Hons) MSc(Med) PhD7
Professor of Public Health, Centre for Health and Social Care Research
Faculty of Development and Society
Centre for Health and Social Care Research
Centre for Health and Social Care Research Sheffield Hallam University, Sheffield, UK
Canada Research Chair in Ethnicity and Health and Associate Professor, School of Nursing, University of Alberta, Edmonton, Alberta, Canada
Professor of Diversity in Health and Social Care, Mary Seacole Research Centre, De Montfort University, Leicester, UK
Professor, Centre for Language and Culture, University of Johannesburg, Johannesburg, South Africa
Received: 17 April 2011
Accepted: 12 June 2011
Researching ethnicity and health presents significant ethical, conceptual and methodological challenges. Although the potential contribution of research evidence to tackling ethnic inequalities in health is recognised, there are widespread concerns about the ethical and scientific rigour of much of this research, and its potential to do more harm than good. The introduction of guidance documents at critical points in the research cycle, including within the peer-review publication process, might be one way to enhance the quality of such research. This article reports the findings from the piloting of a guidance checklist within an international journal. The checklist was positively received by authors and reviewers, the majority of whom reported it to be comprehensible, relevant and potentially useful for improving the quality of published research. However, the rate of participation in the pilot was poor, suggesting that the impact of such a checklist would be very limited unless it was perceived to be an aid to authors and reviewers, rather than an additional burden, and was strongly promoted by journal editors.
• As the volume of ethnicity and health research increases, so too do concerns about its ethical and scientific rigour and its potential to do more harm than good.
• Past attempts by journal editors to raise the standards of published biomedical research on race and ethnicity, although not formally evaluated, appear to have had limited impact.
• A number of factors may undermine attempts to introduce journal guidance for ethnicity and health research, including the diversity of disciplinary perspectives among researchers.
• Our findings suggest that it is feasible to produce a guidance checklist on researching ethnicity that is meaningful and acceptable to a range of health researchers.
Although some authors and reviewers reported that the checklist had a significant effect on their practice, uptake was disappointingly low.
• Journal checklists are unlikely to have a significant impact on research quality unless they are actively promoted by journal editors.
author guidance, ethnicity, peer review, reporting research, research ethics, research methodology
There is now substantial evidence that health and healthcare experiences vary according to ethnicity, and that numerous indicators show that minority ethnic groups tend to be at risk of significant disadvantage across a wide range of settings (e.g. Nazroo, 1997; Gill et al, 2007). The need to understand and tackle such ethnic health inequalities has been repeatedly highlighted in UK policy (Townsend and Davidson, 1982; Acheson, 1998; Department of Health, 2003, 2008) as well as in other European countries and the USA (see, for example, Fernandes and Miguel, 2007; Johnson, 2009). Furthermore, the need for an evidence base that reflects the diversity of the population has been formally acknowledged by, among others, the UK Department of Health in its Research Governance Framework for Health and Social Care (Department of Health, 2005). There are increasing expectations that the generation and application of research evidence can and should play an important role in shaping health policy, practice and debate in ways that can help to ameliorate such inequalities (Petticrew et al, 2004; Williams, 2007; Tugwell et al, 2010).Therefore, although much health-related research continues to exclude participants from minority ethnic groups and/or fails to give considered attention to ethnicity (Mason, 2002; Hussain-Gambles, 2003; Oakley, 2006; Sheikh et al, 2009), research interest in ethnicity and health is growing rapidly in the UK and elsewhere (Drevdahl et al, 2006; Ahmad and Bradby, 2007).
At the same time, just as the volume of research addressing ethnicity and health is expanding, so too are concerns about the quality of this research, its contribution to policy and practice that benefit minority ethnic populations, and its potential role in stereotyping and stigmatising minority ethnic populations (Drevdahl et al, 2006;Gunaratnam, 2007; Bhopal, 2008). Researching ethnicity and health presents significant ethical, conceptual and methodological challenges (Salway and Ellison, 2010). In recent years, for example, health research has been criticized for its tendency to employ untheorised and inappropriate models of ethnicity that present ethnic ‘groups’ as stable, discrete entities (Stubbs, 1993; Bradby, 2003; Aspinall and Chinouya, 2008), for its lack of conceptual clarity and use of imprecise and inconsistent terminology (Bhopal, 2003; Aspinall, 2008), and for inadequately engaging with the multi-dimensional nature of ethnicity (Kaufman et al, 1997). Other commentators have drawn attention to the need for greater consideration of how samples are drawn and how participants are recruited (Nazroo, 1997; Ellison, 2005; Epstein, 2008; Johnson and Borde, 2009), as well as how data are generated from diverse samples, including issues of translation and cross-cultural validity (Atkin and Chattoo, 2006; Johnson and Borde, 2009). Concerns have also been expressed about the paucity of culturally competent research practice, and failure to ensure respect for cultural norms. Without meaningful participant engagement, the interests of minority ethnic research participants are unlikely to be adequately protected (Papadopolous and Lees, 2002; Johnson, 2006; Mir, 2008). Health research has also been accused of focusing disproportionately on rare and exoticised diseases, or behaviours and beliefs that are constructed as deviant and problematic (Ahmad and Bradby, 2007), rather than addressing issues that are of greatest concern to minority ethnic people. More generally, health research has often failed to incorporate a broader social, historical and political analysis of ethnicity, thereby overlooking the power relationships and structural inequalities inherent in ethnic hierarchies (Salway and Ellison, 2010).
Many of these issues have been recognised for some time (see, for example, Colledge et al, 1983), and there have been a number of initiatives to develop standards and introduce guidelines aimed at enhancing the quality of published research. The potential role of journal editors in promoting good research practice in this field has been discussed by a number of commentators (Bhopal et al, 2000; Ellison and Rosato, 2002; Outram and Ellison, 2006a). Indeed a large number of biomedical journals have over the past 10–15 years included editorials that have aimed to alert researchers to potential pitfalls (Outram and Ellison 2005, 2010) and to raise standards (see, for example, the review by Smart et al, 2008 of journal guidance on the classification of race and ethnicity). These forms of guidance have tended to focus primarily on how researchers employ key terms, including race, ethnicity and culture, and how they delineate and label racial or ethnic groups (see, for example, McKenzie and Crowcroft, 1996), as well as promoting the use of anti-racist language. However, a number of other guidance documents have engaged with a wider range of issues. These include detailed guidelines aimed at mental health researchers (Patel, 1999), and the code of the Scottish Association of Black Researchers (SABRE) (2001), both of which are presented as ethical guidance, rather than as guidance on scientific rigour, highlighting the concern that research may be exploitative and perpetuate hierarchies of power and negative stereotypes of minority ethnic people.
Although the development and promotion of such guidance documents would appear to be logical and necessary, given the persistence of the poor research practice highlighted above, to date there has been little exploration of the acceptability of such interventions among researchers, or of their impact on research practice. Ellison and Rosato (2002) concluded that the classification of race/ethnicity in papers published by the BMJ remained haphazard and poorly documented despite the introduction of editorial guidelines in 1996, although it was unclear why the guidelines had not been followed, or how far reviewers had considered these matters when recommending (or not) the acceptance of submissions.
Potential challenges to the promotion and impact of such guidelines within research journals may include the wide diversity of disciplinary perspectives among ethnicity and health researchers, and a consequent lack of consensus on research principles and standards (Salway et al, 2009). Many health-related journal editors need to cater to a multidisciplinary audience of authors and reviewers. This may mean that it is impossible to produce documents that are widely acceptable, or that such documents would need to contain highly flexible prompts rather than prescriptive codes, calling into question whether or not they would actually serve to shift practice (Salway et al, 2009). This observation concurs with the finding of a trend over time, in biomedical journals, away from prescriptive standardisation and towards recommendations that emphasise the need for researchers to carefully articulate and justify their approach to conceptualising and operationalising their concepts and measures (Smart et al, 2008). Furthermore, journal editors may be reluctant to promote adherence to such guidance documents for fear of overburdening researchers and reviewers and thereby disrupting the existing processes of peer review, which largely rely upon the goodwill of unpaid contributors. Finally, a certain degree of scepticism has been expressed in relation to checklists that are intended to enhance ethical and scientific rigour in research in general. There is a suggestion that these can encourage a compliance mentality rather than careful reflection and considered responses to complex issues (Barbour, 2001; Moore, 2006; Outram and Ellison, 2006b). However, it remains unclear whether these factors would preclude the successful promotion of journal guidance.
This paper adds to current understanding of the potential for guidance to enhance the quality of published research about ethnicity and health by reporting on the findings of a pilot exercise that was conducted in a leading international journal, Ethnicity and Health. This journal publishes original papers from a wide range of disciplines concerned with investigating the relationship between ethnicity and health, and it currently has the highest impact factor in the Ethnic Studies Category. The pilot formed part of a larger project focused on ethnicity research in the UK. The aims of the pilot were as follows:
• to assess the feasibility and desirability of introducing a guidance checklist focused on researching ethnicity within the journal
• to gain insight into whether such an intervention could help to enhance the quality of published health-related research that includes a focus on ethnicity.
Following a period of consultation with the journal editors, it was agreed that Ethnicity and Health would host the piloting of a guidance document. Pilots were also conducted in four other journals, including Diversity in Health and Care, although the participation rates were very low, as will be discussed below.
Drawing on a systematically conducted review of published literature focusing on scientific and ethical issues that arise when researching ethnicity, as well as a series of consultations with researchers reported elsewhere (Salway et al, 2011), a draft guidance document was prepared by the core research team. This document was then reviewed and finalised, through a series of iterations, in consultation with the journal editors. The document’s content was therefore agreed by consensus among a small group of active researchers, but represented a synthesis of the key concerns documented in the wider literature. As noted above, one of the objectives of the pilot was to validate this document through a process of assessment by a broader range of researchers. The document was prepared in the form of a checklist (see Table 1) that was intended for use by both authors and reviewers to support the preparation and review, respectively, of original research articles. The pilot was introduced in December 2009 and ran until July 2010. During this period all authors who submitted a paper and all reviewers who were asked to review a paper were sent a standard email that included brief details of the pilot as well as a longer information sheet as an attachment and a link to a short online questionnaire. Participation in the pilot was entirely optional, and participants were made aware that completed guidance checklists would not be reviewed as part of the pilot. Instead authors and reviewers were asked to give feedback on the content, usefulness and appropriateness of the guidance via the online questionnaire, which included both closed and open-ended questions.
During the pilot period, 200 papers were submitted to the journal and 70 reviewers completed reviews. A total of 39 individuals followed the link to the online questionnaire, although only 18 respondents (11 reviewers and 7 authors) completed the online questionnaire in full. One of these respondents was an editor of the journal who wished to gain experience of using the checklist as a reviewer. A further four respondents provided answers to part of the questionnaire. Participants were given the opportunity to access the checklist via the survey if they had not already seen it, or if they wanted to refresh their memory.
During the process of constructing and introducing the guidance checklist a number of issues arose, which illustrate some of the concerns of journal editors and suggest potential barriers to raising ethical and scientific standards.
The first issue concerned the clarity and meaningfulness of the guidance checklist to the journal audience. Thus, although the content of the draft document was not substantially contested, there was a need to tailor the wording and layout to the journal’s own context. Onthe whole this was not a significant issue. However, there were two areas of potential complexity. The first related to the relevance and appropriateness of the guidance checklist to an international readership. This was necessary because although the wider project was focused on the UK, Ethnicity and Health caters to an international and multi-disciplinary body of researchers, authors and reviewers. Although the literature review had suggested that many of the issues raised in the guidance document are recognised cross-nationally, particularly in both the UK and the USA, understandings and terms relating to ethnicity, race and related concepts vary greatly across settings, reflecting particular histories of ethnogenesis (Aspinall, 2007) which demand careful consideration to ensure comprehension and utility. The second issue, which only came to light once the pilot was under way, related to the applicability of the guidance checklist to different types of study. Although the phrasing and content successfully accommodated both quantitative and qualitative empirical studies, a need was identified for some modification to ensure easy applicability to secondary research studies based on the review and synthesis of earlier work. This is an important consideration given the increasingly significant contribution of secondary research to knowledge generation for policy makers and practitioners (see also Tugwell et al, 2010).
The second area of concern for editors related to the potential disruption of normal peer review processes operating within the journal. Details of the pilot were carefully negotiated with the editors. However, there was a concern that the introduction of the guidance might place an additional burden on the editors themselves, and importantly, on their reviewers, who were all unpaid. The solution that was proposed in order to allay these concerns was that participation by authors and reviewers would be entirely voluntary, and the pilot would be hosted rather than actively promoted by the journal. In addition, the length of the checklist was restricted to two sides of A4 paper.
Finally, a major obstacle for the editors related to the technology of the online submission and editing system. Lengthy negotiations with the publishers were necessary to ensure that the checklist and related prompts were made available to authors and reviewers via the automated system.
The responses to the online questionnaire suggested that most respondents considered the content of the checklist to be appropriate, comprehensible and exhaustive (see Table 2). Only one respondent expressed a strong negative attitude towards the checklist, and this related more to the overall desirability of such an intervention than to the content of the checklist itself. We shall discuss this issue in more detail below.
Table 2: Summary of responses from authors and reviewers to questions in the online feedback questionnairea.
A few queries were raised about the potential for misinterpretation of some of the questions and the need to avoid words that were considered to be unusual (e.g. ‘bespoke’ in Question 6). These issues warrant attention in any revised version of the checklist. One respondent stated that Question 19, on reflexivity, was unnecessary. However, it is unclear whether this was because they regarded this topic as part and parcel of all good research, and therefore not something to be highlighted in a checklist focusing on ethnicity, or whether it was because the question itself was in some way inappropriate.
Seven respondents identified some questions that they considered irrelevant and/or difficult to apply to the manuscript in question. However, on examination of their detailed responses, most of the issues that were raised did not suggest the need for changes to the checklist. For instance, one respondent stated that, since all of the study’s respondents spoke the native language, Question 13 was irrelevant and the response option ‘Not applicable’ would have been more appropriate in this case.
Two respondents stated that the checklist was inappropriate for review papers. It is reasonable to argue that the authors of review papers have no control over the methods that are used in the papers that they include. However, the checklist was applicable if it was used in relation to the methods utilised by the authors of the review papers to compile their reviews, and the approach that they adopted when synthesising, interpreting and presenting the findings in these papers. Any revised checklist might, nonethenonetheless, benefit from indicating which questions might only be applicable to specific types of studies or to the methods used by the authors themselves rather than any previous studies that they review.
Only one respondent felt that something had been omitted fromthe checklist. This individual considered that definitions of race, ethnicity and related concepts may vary according to country, and that this should be explicitly acknowledged.
A total of 18 participants provided responses to the questionnaire sections relating to their experience of using the checklist (see Table 2). Again, their responses indicated a generally positive attitude towards the checklist, with just three out of 18 respondents reporting that use of the checklist took ‘too much time’, and only one stating that it made the task of preparing or reviewing the manuscript ‘more difficult.’ When asked whether using the checklist had had a significant impact on the way in which they had reviewed or prepared the manuscript, seven out of 18 respondents gave a positive response (3 out of 7 authors and 4 out of 11 reviewers). Comments included the following:
‘Helps in identifying important issues when writing a paper on race/ethnicity.’
‘By using the checklist it is easier to review the concept of ethnicity in the study in a systematic way.’
With regard to the authors and reviewers who did not consider the checklist to have had a significant effect on their own behaviour, this was primarily because the checklist was felt to cover issues that they would normally take into consideration anyway. Although their answers suggested that the checklist had not affected their own behaviour, their responses nevertheless offered a positive endorsement for the checklist content. For example:
‘I used already the same principles in my scientific research.’
‘Most of the issues raised in the checklist are things I would normally be attentive to in a review.’
When asked ‘Do you think that the checklist can enhance the quality of the papers published in the journal?’, all 18 respondents gave a positive response. Further scrutiny of what the respondents had written in the open-ended answer format here revealed a variety of ways in which the respondents felt the checklist would help to enhance the quality of published papers. These included the following:
• raising awareness among researchers:
‘It may be a way of educating authors and reviewers about ethical considerations involved in this type of work.’
‘[It] covers many important issues that authors may have neglected.’
• contributing to the rigour of research and the systematic reporting of studies:
‘It will help to create an international standard in the concerned scientific literature.’
‘If researchers use the checklist when preparing manuscripts, then a more consistent and focused treatment of racial and ethnic issues should be the result.’
‘I think it is a good move towards research quality.’
• assisting reviewers in their task and making reviews more useful and standardised:
‘The checklist encourages the reviewer to really think carefully before reading the paper about these issues, so when you are reading the paper you are looking for these criteria in the paper.’
‘It gives a nice framework for issues to consider when reviewing, in one easy-to-access place.’
However, it should be noted that one author expressed the view that checklists should not ‘be used to direct research’, and three respondents highlighted the possibility that checklists may represent a burden and constraint for researchers, particularly if they are too long:
‘I think that the checklist is still too long and should, ideally, be boiled down to something snappier.’
‘Applying a checklist may be overly burdensome and constraining.’
The objectives of the piloting exercise described above were first to assess the feasibility and desirability of introducing a guidance checklist on researching ethnicity within the journal, and secondly to gain an insight into whether such an intervention might help to enhance the quality of published health-related research that includes a focus on ethnicity. Before summarising the findings and drawing conclusions, it is important to highlight three limitations of the study. First, the pilot only ran for a short period of time, and thereforemay not predict the experience of embedding a guidance checklist into a journal’s processes over an extended period of time. Secondly, uptake of the guidance by authors and reviewers was entirely optional, so the findings may not be predictive of the outcomes and impacts of a more actively promoted or mandatory intervention. Thirdly, the number of responses to the online questionnaire was low, which is both a finding and a shortcoming of the study.
Despite these limitations, the feedback from the authors and reviewers who participated in the pilot showed a predominantly positive response to the checklist. The checklist was thought to be comprehensive, exhaustive, relevant and useful by most of the respondents, confirming the feasibility and perceived desirability of the intervention.
Only one respondent expressed strong negative views about the checklist, which included concern that it could stifle researcher creativity. In addition, a small number of respondents identified refinements that would further improve the checklist, including increasing its relevance for review papers, improving its comprehensibility and applicability to an international and multidisciplinary audience, and reducing and/or optimising its length, although it was unclear which specific questions the respondents thought could be omitted.
With regard to the checklist’s potential to enhance the quality of published papers, the respondents again painted a predominantly positive picture. The checklist was reported to have had a significant impact on manuscript preparation and/or review for a number of respondents. Meanwhile, for those respondents who reported no significant impact, this was primarily because the checklist was felt to be consistent with their current practice, rather than because it was unhelpful, inapplicable or inappropriate. Indeed, all of the respondents felt that the checklist could help to enhance the quality of papers published in the journal, identifying a range of benefits that would flow from its use.
However, it is important to consider the extent to which the respondents were a self-selected group who might have already been positively predisposed towards the guidance. Around 50% of the respondents described themselves as experienced reviewers or authors of journal papers, and a further eight rated themselves as intermediate. Only one respondent indicated that they were a novice. In addition, six respondents indicated that they were experienced and eight rated themselves as intermediate with regard to the area of ‘race’, ethnicity and/or minorities research, while four described themselves as novice. This is somewhat reassuring, as the range of expertise among pilot respondents means that important inadequacies in the checklist would have been highlighted if they had existed, and also that the checklist was considered to be useable by a range of researchers with varying levels of previous exposure to the issues covered.
However, it is still possible that our respondents were more interested in the area of research standards than the average reviewer or author (or at least the much larger number of authors and reviewers who did not respond to our invitation to participate in our pilot study). All 18 respondents who completed the entire questionnaire reported generally favourable opinions of journal guidance, although several qualified their responses, identifying both advantages and drawbacks. Our results might therefore suggest a more positive response from the pilot participants than would be the case for the journal’s wider audience of authors and reviewers. Unfortunately, we can only speculate about this, as we were unable to obtain any comparable data from non-respondents, or information about the reasons for their non-participation in this pilot study.
Nevertheless, the low level of participation in the pilot warrants careful reflection, and however useful the checklist might have been to those who chose to use it, its impact will be minimal if it is adopted by such a small proportion of authors and reviewers.
Bearing this in mind, it is worth considering whether there might have been a higher level of participation by authors if there had been some mechanism to ensure that they had access to the checklist prior to submission. Unfortunately, this was not possible. Instead, a large proportion of the authors who were eligible to participate in the pilot study would only have become aware of the checklist at the time when they submitted their manuscripts. Under these circumstances, most authors may well have preferred to continue without taking the additional time to consider the guidance and make any necessary adjustments. The low response rate among reviewers is more disappointing. The pilot information was emailed to all of them when they were invited to review. The lack of response may corroborate concerns expressed by the editors during the pilot design that reviewers already feel overburdened and are reluctant to engage in something that is perceived as increasing their workload further. Given that Ethnicity and Health has an explicit focus on ethnicity, and one would assume that authors and reviewers have an interest in this field, we had expected the participation rate to be higher. It may be that contributors to the journal who chose not to participate made this decision because they felt that they were already well versed in the issues concerned, or because they had little interest in such efforts to improve the quality of published research. It is important to note that participation rates in pilots that were conducted in four other more generic social science journals were even lower, which suggests a lack of engagement and/or interest in the issues addressed in the guidance.
If the checklist is to be more widely adopted, it would therefore seem to be important to find ways of presenting it as an aid to reviewers and authors that makes their job easier, rather than as an additional task for them to complete. Moreover, high levels of uptake would appear to be unlikely if the checklist is presented as being optional, rather than being actively promoted by editors. However, even under these circumstances, offering reviewers the option of using such guidance may still be a useful first step towards improved and consistent practice, given that many of the respondents felt that it had either helped them to address the issues raised or confirmed their existing practice. As such, although our findings suggest a rather limited impact on research quality in the short term, such guidance might be expected to incrementally improve practices among authors, reviewers and editors over time if it was to be embedded in the journal’s processes over the longer term. Furthermore, it is to be hoped that if manuscripts were referred for revisions or rejected completely on the basis of issues raised in the guidance, researchers might be encouraged to adjust the design and conduct, as well as the reporting, of their research. Editors of journals that carry greater weight with authors, reviewers and publishers may be in a stronger position to promote, or even make mandatory, the adoption of such guidance and thereby contribute to the enhanced quality of published research on ethnicity and health. Furthermore, as the volume of research addressing ethnicity and health expands, it will be important for mainstream journals to engage seriously with the issues of scientific and ethical rigour that have long been of concern to more specialist researchers in this field.
The project upon which this paper is based was funded by the Joseph Rowntree Foundation. We thank Helen Barnard of the Foundation and members of the Project Advisory Group for their valuable input to the project. We also thank Hannah Bradby, Karl Atkin and Seeromanie Harding, the editors of Ethnicity and Health, for their support with regard to the pilot work, and the authors and reviewers who participated in the pilot.
