Quality in Primary Care Open Access

Keywords

Significant event analysis; Human factors; Patient safety; Primary health care; Incident investigation

Abbreviations

SEA: Significant Event Analysis; NES: NHS Education for Scotland; NHSAA: NHS Ayrshire and Arran

Introduction

Recent evidence has suggested that between 2-3% of general practice (GP) consultations may include a patient safety incident – an event that caused harm, or had the potential to cause harm [1]. Learning from these incidents by using significant event analysis (SEA) is a well-established educational, patient safety and quality improvement tool in many primary health care services worldwide [2,3]. However, there is evidence that SEA is often not performed effectively by care teams and is a missed opportunity for learning and improvement [2,4]. A key barrier to SEA effectiveness is that the ‘cause’ of the event is often attributed to ‘human error’ with a lack of a ‘systems approach’ underpinning the analytical process in line with latest safety science thinking and consensus [5-7]. This limited approach can have multiple negative impacts including, for example, emotional impacts on the care practitioners involved (the socalled ‘second victim syndrome’ [8], leading to the failure to implement robust, sustainable change to reduce the risk of occurrence of the event [7].

To address these deficiencies a process for undertaking SEA more effectively - based largely on Human Factors principles - has been developed and tested (termed ‘enhanced SEA’) [7]. This approach aims to help care practitioners and teams reflect on the personal impacts of the event and adopt a ‘systems approach’ to the analysis to provide better insights into why things have gone wrong and so direct more meaningful improvements.

NHS Ayrshire and Arran (NHSAA) is one of 14 regional health boards in the Scottish Health Service. NHSAA encouraged the participation in localised safety and improvement work of general practice (GP) teams through an optional, financially incentivised, patient safety enhanced service. The specifications for the 2015-2016 patient safety enhanced service included completing and submitting an enhanced SEA to NHS Education for Scotland (NES) for peer review using a well-established system [5]. NES is the national body with responsibility for the education and training of the healthcare workforce and is active in safety and improvement development and research. Participating GP teams were directed to submit enhanced SEA reports which described the analysis of a patient safety incident.

The aim of this study was to thematically summarise the types of significant events selected for analysis and submitted by local care teams and the agreed improvement actions. The quality of submissions, as judged by the trained NES peer reviewers, was also compared with those of other professional groups.

Methods

Participants, setting and incentivisation

All 55 general practice teams in NHSAA region were invited by the health board to participate in the ‘enhanced service’ on a voluntary basis. Participants were incentivised as part of local enhanced contractual arrangements and received additional ‘back-fill’ funding for attending related training and subsequently undertaking and submitting enhanced SEA reports.

Enhanced SEA training sessions

Participating practices were required to send two representatives, one of whom was a GP, to attend two halfday patient safety training sessions. These were designed and facilitated by two local GPs who were also the organisational clinical leads for patient safety in NHSAA. Both are experienced medical educators and researchers and were also trained in the enhanced SEA method. During each training session, 1 h was dedicated to learning focused on the enhanced SEA method. This included presentation on the theory of adopting a Human Factors approach to SEA, describing the enhanced SEA methodological process and participating in simulated analyses of example significant events (including their own events) as part of small group work which were assisted by a GP facilitator (Table 1).

Table 1: Delivered enhanced SEA training topics and educational modality.

Identification and analysis of Significant Events and submission of reports

GP teams identified ‘significant events’ for analyses during the study. Each team was required to analyse a single event as a practice team and apply the principles and approaches previously taught. Each participating practice were required to submit one enhanced SEA report to NES for educational peer review [5] within two months of the second training interventions. Participants were made aware that fair, constructive and developmental feedback would be provided by independent, external and trained GP peer reviewers. Enhanced SEA reports were completed using standardised pro-forma provided to teams and submitted via secure electronic mail to NES [9].

Peer review system and underlying theoretical principles

NES operate a well-established system for the peer review of SEAs based on sound educational principles that has previously been described and evaluated [5] (Figure 1). The theoretical approach underpinning the system is based on an adaptation of cognitive continuum theory. This helps to understand and frame the thinking used in performing different tasks and to attempt to improve the level of reflection on these tasks. How each task is accomplished is described using one of six ‘modes of practice’ that range from highly structured scientific experiment (mode 1) to intuitive judgements (mode 6) with peer review positioned between modes 4 (system-aided judgement) and 5 (peer-aided judgement). The peer review approach aims to improve the conclusions and decisions from the analysis by minimising the utilisation of mode 6 judgements (self-assessment) [10,11].

Figure 1: NHS Education for Scotland SEA peer review process.

Submission of an SEA for peer review is a requirement of work placed based assessment for GP specialty trainees in the west of Scotland and for GPs who wish to train to become educational supervisors. The system is open to all qualified GPs on payment of a fee, however, Ayrshire and Arran GPs were able to submit SEA reports without cost.

The peer review team consists of qualified GPs (n=25) and a lead peer review co-ordinator who are trained and regularly calibrated in the SEA review and feedback processes by NES medical educators. Submitted SEA reports are annonymised prior to review although details of whether the report was submitted by a GP specialty trainee or a qualified GP are included. Reports are reviewed by two members of the peer review group who complete a feedback instrument [12] that includes grading the project (on a scale from 1=very poor to 7=outstanding) and giving formative feedback in several domains (Table 2). The grades given by peer reviewers in each domain listed in Table 2 are routinely inputted and stored in an electronic database by NES. A letter is generated by the lead co-ordinator based on the reviewers’ comments to give formative feedback to the author on the quality of event analysis in each of seven domains (the description of the event, its actual or potential impact, the exploration of contributing factors to why the event occurred, learning and reflection on the event, involvement of appropriate individuals, the action plan and the overall conclusion) and how this could be improved, where judged necessary, or provide reassurance and validation of good practice. GP teams receiving the feedback letters are under no obligation to act on this guidance, although the assumption is that feedback from two independent GP colleagues on an important safety issue will prompt action if considered to be relevant and feasible.

Table 2: SEA feedback instrument domains.

Data analysis

Each enhanced SEA was analysed independently by an NHSAA patient safety clinical lead (DM) and a NES safety and improvement project manager (PW). The involved practice system and the actions implemented were recorded using a previously published data collection framework [5]. This was adapted iteratively to refine coding as the study progressed. Due to the complexity of some events many different care systems were involved. A pragmatic approach was taken to assign each report to the dominant system involved. For example, a prescription was issued for a live vaccine for a patient who was immune-compromised. Although a prescription was issued this event related more to the vaccination protocol rather than the prescribing system.

The involvement of patients in the event analysis, either directly or by explicitly obtaining and presenting their perspective, was recorded. The National Reporting and Learning Service (NRLS) risk matrix was used to grade the actual consequence of each event as described in the report using a five-point scale (where 1 - minor, 2 - one for negligible, 2 - minor, 3 - moderate, 4 - major and 5 - catastrophic) [13].

The grade given in each domain to NHSAA reports were compared with two other cohorts, GP specialty trainees and GP prospective trainers submitting SEA reports for peer review over the same period (31^st March 2015 to 31^st March 2016).

Data were analysed using simple described statistics such as frequency counts, percentages, means and 95% confidence intervals.

Results

Study participation rate

A total of 51 NHSAA GP teams voluntarily participated in the enhanced service (51/55, 93.0%), attending the training sessions and submitting 52 enhanced SEA reports for peer review.

Main type of incident and examples

Significant events described in submitted event analysis reports were classified and themed across five main sub-systems (e.g. prescribing system and test result handling system). Table 3 shows the number of projects and examples of projects submitted in each system (Table 3).

Table 3: SEA incident system domains with frequency and examples of incident type.

Actions implemented or planned

A total of 113 implemented actions for system improvement were described in the 52 projects (mean 2.2 per project). This did not include discussion of the event within practices as this was required prior to submission of the reports. Eight themes of actions implemented were agreed. These are shown in Table 4 along with the frequency of this action (Table 4).

Table 4: Overall frequency of actions in each domain.

The frequency of described actions in each of the main system domains are described in Table 5. The most frequently reported action was alteration of practice protocols. Overall, 35% of reports described changing protocols and 19% described educational interventions. Interventions that relied on individual action only, for example increased vigilance, was reported in only five percent of reports. Projects were infrequently discussed with other areas of health and care. Only seven of 12 SEAs that specifically related to interactions between general practice and other area of health and care were discussed with professionals outside the practice (58%).

Table 5: Reported improvement actions implemented in each incident system domain.

Patient involvement

Two of the submitted reports (3.8%) described direct involvement of the patient in describing their perspective on what happened, the impact they experienced or in the design of the implemented change. Examples are the design of new systems to communicate results to patients and the design of a new system for INR monitoring.

Grading of consequences

The grading of the consequences of the events is shown in Table 6. The majority of the enhanced SEA reports described events whose consequences were negligible or minor (86.6%). Five events had consequences that were moderate or severe (9.6%).

Table 6: Grading of risk consequences of SEA reports (n=52).

Quality of SEA report

The mean grading given in each domain of the SEAs by NES peer reviewers is shown in Figure 2. Comparison is made between NHSAA SEA reports, those completed by prospective GP educational supervisors (n=38) and those completed by GP specialty trainees (n=98). All three groups have similar mean scores in all domains. The actions described in Ayrshire and Arran SEAs were graded lower than those of prospective trainers but higher than those of GP specialty trainees. The differences did not reach statistical significance.

Figure 2:Mean grading given by peer reviewers with 95% confidence intervals (1- very poor, 2- poor, 3- fair, 4- good, 5- very good, 6- excellent, 7- outstanding).

Discussion

This study examined enhanced SEA reports submitted by frontline GP teams in a single health board area as part of an incentivised regional initiative to improve learning and action from the analysis of patient safety incidents. The main findings suggest that participants were largely able to undertake ‘systemscentred’ event analyses as judged by external peer review.

Types of incidents and their consequences

A relatively large proportion of incidents described communication problems with external care services and a higher proportion of events with lower consequences were reported than have previously been described [5]. As reports were submitted to the local health authority, low consequence events may have been purposively selected so that GP teams were less likely to be seen to be at fault. Furthermore, the submission of events where communication with external care sectors was problematic may have been prioritised in an attempt to emphasise the potential for harm at care interfaces and to stimulate improvement efforts at a Health Board level. The evidence base reports that patient safety incidents related to communication issues is one of the most common contributory factors in why things go wrong [14]. The nature of communication problems between primary care and other care sectors often revolves around issues to do with transfer of accurate information in a timely manner and many of these challenges and their impacts were described the submitted reports [15].

Incident reporting

Although incident reporting is well established in secondary care, reporting rates in primary care in the UK and internationally are very low, to the extent that routine engagement with formal reporting systems is virtually non-existent for many if not most teams – a circumstance mirrored in NHSAA [16]. Submission of reports as part of this study may have been considered a form of incident reported by teams; however, less than 15% of event submissions in this study stated that the event had been formally and independently reported to the health board. Insufficient time, lack of feedback, fear of blame, perceived reputational damage and patient confidence have been found to be barriers for reporting of incidents by GPs [17]. Adoption at all levels of health care of key principles for understanding and responding to patient safety incidents (a fundamental learning needs for care practitioners, educators and policy makers) may help overcome some of these barriers [18,19].

Actions implemented

Rather than the propensity to frequently blame individual personal and professional failings (self and others), the adoption of a ‘system approach’ to inform more constructive and meaningful organisational learning following adverse events has been recommended in influential policy documents in the UK such as the ‘Organisation with a Memory’ report and more recently, the ‘Francis report’ [20,21]. In a recent study evaluating actions after a different incident investigation technique, Root Cause Analysis, the most common reported action was training [22]. It is encouraging that SEAs in this study often described actions aimed at improving a wide range of care systems and procedures rather than focussing on being more vigilant or undertaking personal education or refresher training which are viewed as limited or weak improvements as part of the hierarchy of risk management interventions [23,24].

Many of the SEAs that centred around clinical decision making issues reported educational changes but all of these events also included changes to practice systems, which is gratifying given that ‘diagnostic error’ and related cognitive challenges are known to be influenced by wider system factors [25]. It may be that rather than practitioners blaming themselves or colleagues and proposing to ‘try harder’ or learn more about a specific area, enhanced SEA has encouraged a ‘system approach’ and potentially more robust change ideas as a consequence, particularly compared with the previous research in this area [5,22].

There was limited evidence that SEA reports described discussions with other areas of health and care. Even when the event involved interactions with other professions just over half were discussed with these professionals. This may represent difficulties in contacting those involved, time available to contact them or lack of belief that such interactions will be worthwhile [17]. Practices may have submitted SEAs describing interface issues with other parts of health and care in the hope that the act of submission will lead to change. Further research is needed to explore this in depth.

The design or redesign of practice protocols and procedures were described as the most frequently occurring actions for improvement in all system domains. This demonstrates to some extent the complex and imperfect nature of these care systems; an issue that goes to the heart of most patient safety incidents in general medical practice and wider healthcare [26]. It also highlights a number of challenges around the design and implementation of practice procedures as barriers in safety management. The implementation of protocols is often seen as a ‘robust change’ however in the complex sociotechnical systems present in health care it can be difficult to specify work adequately to cover all conditions faced by staff. Work-asdone by frontline staff in order to achieve success often has to deviate from work-as-imagine in protocols. Obtaining multiple perspectives of staff on proposed changes and designing new protocols in a participatory manner may be more likely to result in work-as-done being closer to work-as-imagined [27].

Patient Involvement and early resolution of complaints

Involvement of patients in SEA is encouraged but has been reported as being difficult to achieve [28,29]. In this study only a small minority of practices reported the involvement of patients and families. Scottish and UK legislation will soon come into force regarding the organisational expectations around discussion of safety incidents with patients and families as part of a Duty of Candour – the professional expectation is already embedded in good clinical practice by regulators [30,31]. Further guidance and training on how this can be achieved more readily is needed if this is to be adopted more widely as part of a person-centred approach to quality care.

GPs have reported that completing SEA may be useful when a complaint has been received as this may demonstrate to patients that it has been taken seriously [32]. Indeed, it may be that the use of SEA may aid the early resolution of complaints [33]. There are potential barriers to reflecting openly when analysing significant events as reports could be requested by a court of law [34]. Focusing on systems based learning rather than blaming individual actions may be a helpful approach.

Quality of reports

Measures of quality of SEA reports between Ayrshire and Arran GPs, prospective trainers and specialty trainees were similar. This demonstrates that after training frontline GP teams can perform SEA analyses of equal quality to previously identified early adopters and career motivated individuals. The quality can be improved further and the development of education interventions to enhance improvements is required.

Strengths and limitations

This study showed high engagement in a single health board and used a well-established and evidence based educational system but there were many limitations to this study. The numbers were small and so there is limited generalisability. Moreover, the type of incident analysed and submitted may have been influenced by the fact that this was part of an enhanced service. The intervention required the provision of dedicated training time that may not be possible for most GP teams across the UK. Ayrshire and Arran employed two GPs who had expertise and in enhanced SEA who were able to deliver the training. This approach may not be feasible for all health boards.

Implications for policy, practice and future research

To improve the quality of SEA reports in primary care and thus improve the learning and the effectiveness of change, priority has to be given to developing the knowledge and skills of frontline GP teams to perform SEA and the protection of time to complete SEA. For some health boards, capacity will need to be developed to train frontline teams.

Improving relations across care interfaces may be required before joint analysis of events can be undertaken but consideration needs to be given to how best to facilitate the involvement of other care sectors in analyses [35]. This may include deciding on the best method of communication (telephone, electronic mail or through existing reporting systems), guidance on how communication should be conducted (for example by trying to understand why decisions were made in other areas rather than apportioning blame) and ensuring relevant feedback is given after analysis [36]. If incident reporting in primary care is to be prioritised at a national level, GPs will need time and training on how to report events and which events to report. Further research into the barriers to patient involvement in SEA is also required if this area of SEA is to be developed.

Conclusion

This study has demonstrated that most GP practices, after minimal training, can apply the enhanced SEA method and largely adopt systems based interventions to improve care by analysing and respond to past harm or near misses. Practices described integrating learning to improve care systems which is a key component of Vincent’s Framework for measuring and monitoring safety [37]. However, to be more comprehensive, consideration of how to improve the analysis of events across healthcare interfaces, involve patients to maximise learning, increase incident reporting and employ prospective hazard analysis methods, are also needed. Although this is a comparatively small study, it provides some important evidence that this ‘new’ approach is leading to more effective and meaningful event analyses by primary care teams - thereby potentially overcoming the key criticisms of the ‘old’ personfocused method which encouraged a limited, superficial and non-systematic investigation of a healthcare problem or issue.

Funding

Practices received funding from Ayrshire and Arran health board to participate in the enhanced service. No other funding was received for this evaluation.

References

1. Panesar SS, de Silva D, Carson-Stevens A, Cresswell K. How safe is primary care? A systematic review. BMJ Qual Saf 2016; 25: 544-553
2. Bowie P, Pope L, Lough M. A review of the current evidence base for significant event analysis. J Eval Clin Pract 2008; 14: 520-536.
3. Scott I. What are the most effective strategies for improving quality and safety of health care? J Intern Med 2009; 39: 389-400.
4. McKay J, Bowie P, Lough JRM. Variation in the ability of general medical practitioners to apply two methods of clinical audit: A 5 year study of assessment by peer review. J Eval Clin Pract 2006; 12: 622-629
5. McKay J, Bradley N, Lough M, Bowie P. A review of significant events analysed in general practice: Implications for the quality and safety of patient care BMC Fam Pract 2009; 10: 61.
6. Dekker S. Safety differently: Human factors for a new era. CRC Press, Boca Raton 2015.
7. Bowie P, McNaughton E, Bruce D. Enhancing the effectiveness of significant event analysis: Exploring personal impact and applying systems thinking in primary care. J Contin Educ Health Prof 2016; 36: 195-205.
8. Dekker S. Second victim: Error, guilt, trauma and resilience. CRC Press, New York 2013.
9. https://www.qihub.scot.nhs.uk/media/1072905/sea%20report%20form_new16.pdf
10. Hammond KR. Towards increasing competence of thought in public policy formation. Judgement and decision in public policy formation. Westview Press, Boulder, CO 1978; 11-32.
11. Bowie P, Cameron N, Staples I, McMillan R. Verifying appraisal evidence using feedback from trained peers: Views and experiences of Scottish GP appraisers. Br J Gen Pract 2009; 59: 484-489.
12. McKay J, Murphy DJ, Bowie, P Schmuck ML. Development and testing of an assessment instrument for the formative peer review of significant event analyses. Qual Saf Health Care 2007; 16: 150-153.
13. https://www.nrls.npsa.nhs.uk/EasySiteWeb/getresource.axd?AssetID=60149&
14. de Wet C, Black C, Luty S, McKay J, O’Donnell C. Implementation of the trigger review method in Scottish general practices: Patient safety outcomes and potential for quality improvement. BMJ Qual Saf 2016; 0: 1-8.
15. Evidence scan: Levels of harm in primary care. The Health Foundation, London 2011.
16. O'Beirne M, Sterlin PD, Zwicker K, Hebert P, Norton PG. Safety incidents in family medicine. BMJ Qual Saf 2011; 1-6.
17. Rea D, Griffiths S. Patient safety in primary care: Incident reporting and significant event reviews in British general practice. Health Soc Care Commun 2016; 24: 411-419.
18. Evidence scan: Complex adaptive systems. The Health Foundation, London 2010.
19. McNab D, Bowie P, Ross A, Morrison J. Understanding and responding when things go wrong: Key principles for primary care educators. Education for primary care 2016; 27: 258-266.
20. An organisation with a memory. Report of an expert group on learning from adverse events in the NHS chaired by the Chief Medical Officer, Stationery Office, London 2000.
21. The Mid Staffordshire, NHS Foundation Trust Public Inquiry. Report of the Mid Staffordshire NHS Foundation Trust Public Inquiry. The Stationery Office Limited, London 2013.
22. Kellogg K, Hettinger Z, Shah M, Wears R. Our current approach to root cause analysis: Is it contributing to our failure to improve patient safety? BMJ Qual Saf.
23. https://www.visualexpert.com/Resources/safetyhierarchy.html
24. https://www.ismp.org/newsletters/acutecare/articles/19990602.asp
25. Trowbridge RL, Dhaliwal G, Cosby KS. Educational agenda for diagnostic error reduction. BMJ Qual Saf 2013; 22: ii28–ii32.
26. Evidence scan: Improving safety in primary care. The Health Foundation, London 2011.
27. Hollnagel E. Safety-I and safety-II the past and future of safety management. Ashgate, Surrey 2014.
28. https://www.nrls.npsa.nhs.uk/EasySiteWeb/getresource.axd?AssetID=61501
29. McKay J, Bowie P, Murray L, Lough M. Levels of agreement on the grading, analysis and reporting of significant events by general practitioners: A cross-sectional study. Qual Saf Health Care 2008; 17: 339-345.
30. https://www.parliament.scot/S4_Bills/Health%20Tobacco%20Nicotine%20etc.%20and%20Care%20Scotland%20Bill/b73s4-introd.pdf
31. https://www.gmc-uk.org/static/documents/content/GMP_.pdf
32. Bowie P, McKay J, Dalgetty E, Lough M. A qualitative study of why general practitioners may participate in significant event analysis and educational peer assessment. Qual Saf Health Care 2005; 14: 185-189.
33. 33.https://www.scottishhealthcouncil.org/publications/research/listening_and_learning.aspx#.WMWmPjvyiUk
34. https://www.aomrc.org.uk/wp-content/uploads/2016/11/Academy_Guidance_on_e-Portfolios_201916-5.pdf
35. Sampson R, Barbour R, Wilson P. The relationship between GPs and hospital consultants and the implications for patient care: A qualitative study. BMC Fam Pract 2016; 17: 45.
36. MacRae C. The problem with incident reporting. BMJ Qual Saf 2015; 0: 1-5.
37. The measurement and monitoring of safety. The Health Foundation, London, 2013.