Quality in Primary Care Open Access

Keywords

appraisal, CPD, learning, practice

Introduction

The principles for implementation of revalidation are based on the proposals in the government White Paper Trust, Assurance and Safety.[1] A key principle is that revalidation will depend on the quality, consistency and nature of appraisal.[2] Annual appraisal for all doctors was recommended by the Chief Medical Officer (CMO) to tackle poorly performing doctors, and was aimed at protecting patients and improving quality standards.[3] The NHS Plan introduced the contractual requirement for all GPs to be on the list of a health authority and be subject to clinical governance arrangements, including an annual appraisal.[4] There were no prescribed guidelines for the implementation, so primary care trusts organised individual programmes; a national electronic toolkit has been developed – this has encouraged uniformity in structure but there are likely to be local individual variations. [5]

Speculation has surrounded the future links between appraisal and revalidation and the role of appraisal as either a formative developmental tool or as a summative tool in revalidation. The NHS Plan stated that the regulation of the profession and individual clinicians needed to be strengthened. Annual appraisal would underpin this and provide data to support the proposed mandatory revalidation process by the General Medical Council (GMC).[6] The report of Dame Janet Smith led to the postponement of revalidation as she concluded that the proposed system lacked rigour, was not a true evaluation of performance and that appraisal was an ineffective method of detecting doctors who are delivering care to a poor standard. The proposal was made to move from a formative to a summative, audited appraisal process to provide a more rigorous objective process.[7] In contrast, the CMO Sir Liam Donaldson, in his later report Good Doctors, Safer Patients, described the appraisal system as a sound process with benefits which should not be overlooked. He also stated:

There is disagreement about whether it should ever have a ‘summative’ element or should it remain as a ‘formative’ tool to enhance learning, improve practice, or drive continuing professional development.[2]

It was recommended that NHS appraisal be standardised, to make judgements about performance against standards, in order to make the process more consistent. [2] More definitive guidance has been provided in Medical Revalidation: principles and next steps [8] which details a two-strand process of relicensing and recertification. This will rely on annual locally based appraisal informed by other evidence including periodic multisource feedback. The appraisal process needs to reflect the diversity of practice and will include a standardised module derived from the GMC’s Good Medical Practice.[9]

In contrast, less has been written about the impact of the appraisal process on the appraisee.However, the strength of the formative developmental process for encouraging the learning process and the role of the PDP at the heart of the appraisal process has been recognised.[10] There is little causal evidence available to demonstrate a strong link between the provision of an appraisal process and an improvement in practice; further research is needed in this area. However, models of appraisal and the critical success factors in delivering GP appraisal have been studied; results have highlighted the importance of the dynamics between appraisee and appraiser and demonstrated that shared goals could result in a rewarding, constructive appraisal. [11]

The majority of research to date which has explored experiences of the appraisal process has been carried out using qualitative methods and small sample sizes which did not allow the results to be generalised to a wider population. A literature search prior to this study revealed little quantitative data on the outcome of appraisal in primary care; this provided an impetus to conduct a local regional survey utilising a quantitative method and incorporating qualitative elements to obtain a broader view of appraisal experiences from a larger sample of GPs. The local approach to appraisal was to encourage all doctors to use the electronic toolkit to reflect on and document their clinical practice, the methods by which they maintain their learning, their relationships with both colleagues and patients, the importance of probity and to help formulate their thoughts as they individually wrote a PDP.

The aim of this study was to investigate and explore GPs’ experiences of the current appraisal process. Specific objectives were to consider the impact the appraisal process had on the learning, practice and CPD of GPs, as these were areas which had been highlighted in Good Doctors, Safer Patients.[2]

Methods

A cross-sectional, descriptive survey design utilising postal questionnaires was conducted in the West Kent region. The questionnaire was distributed to the total population (n = 385) of GPs in West Kent who had participated in the appraisal process in the previous 12 months and elicited a response rate of 71.7% (n = 276). The questionnaire contained structured and open questions on demographic and employment information together with further questions exploring the impact and benefit of appraisal on learning, practice and CPD. Structured questions were designed to measure attitudes, perceptions and views; responseswere recorded using a Likert level of agreement scale. Open-ended questions allowed respondents to expand on reasons for their scaled answers at length and in their own words.[12]

Content validity and utility of the questionnaire were established by an expert panel of experienced appraisers (n = 5) who commented critically on the clarity, layout, length, relevance and content of questions. Changes were subsequently made to wording, layout and format of the questions. Further development pilot work tested the degree of consistency and repeatability (test–retest reliability) of the questionnaire as a data collection instrument, with a sample group of volunteer GPs (n = 6) chosen from the appraiser group. Percentage agreement across all items was determined at 69% when the questionnaire was administered twice at an interval of two weeks.

The Kent Primary Care Agency (KPCA) independently distributed the questionnaire to all the doctors on theWest Kent performers’ list, with a covering letter explaining the study purpose and ethical requirements for anonymity and confidentiality; a stamped addressed envelope was enclosed. To maximise response rates, four weeks later a further mailing was sent to all participants (in view of the requirements for anonymity) with instructions to ignore the questionnaire if they had already completed it.

Quantitative data were entered on SPSS and analysed using descriptive, non-parametric statistics. Content of responses to each of the open questions were analysed using the thematic approach of Hays.[13] Intensive review of the content enabled categories to be coded and recurrent themes to be identified.

Results

Results of the analysis of questionnaire responses are presented in three sections: (i) tables summarising quantitative response frequencies for categorical data; (ii) qualitative findings relating to core themes (encouragement, documentation, reflection, relationship between appraisal and adult learning, external drivers, productivity and time issues in appraisal) which emerged from the thematic analysis of responses to open questions; (iii) qualitative findings relating to core themes (purpose of appraisal and links with revalidation, the appraisal process as an educational tool, the role of the appraiser, the effect of the appraisal process on time and use of resources) which emerged from thematic analysis of respondents’ suggestions for changes to the current approach to appraisal.

As shown in Table 1, the majority of respondents worked in group practices, were practice principals and in full-time employment. Figures from KPCA stated that 57% of GPs were male and 43% female. In this study 59.8% of respondents were male, 23.6% female and 16.7% did not declare gender on their returned questionnaires. It is therefore possible that GPs of female gender were under-represented in the sample with reference to the regional population.

Experiences of taking part in appraisal and the impact on the learning process were encapsulated in responses to three questions, results of which are summarised in Table 2. More than half of the respondents (overall n = 148; 53.6%; proportionally more non-principals than principals) agreed that the appraisal process had affected the learning process whilst fewer respondents (n = 49; 17.8%) disagreed with this. Taking part in the appraisal process had enhanced learning for many (overall n = 131; 47.5%; proportionally more nonprincipals than principals), whilst fewer respondents (n = 71; 25.7%) disagreed with this.Although a minority of respondents (n = 11; 4.0%) reported that appraisal had discouraged their learning, the majority (n = 191; 69.2%) disagreed that appraisal had discouraged their learning.

Respondents were asked to consider the impact of the appraisal process on their practice. As shown in Table 2, changes in practice were reported by fewer than half of the respondents (overall n = 107; 38.8%; proportionally more non-principals than principals), whilst almost one-third (n = 79; 28.6%) disagreed that appraisal had exerted an impact on practice. Many respondents (overall n = 111; 40.2%; proportionally more non-principals than principals) agreed that there had been an improvement in their practice as a result of the appraisal process, whilst fewer (n = 74; 26.8%) disagreed. Negative effects on practice were reported by a minority of respondents (n = 18; 6.5%); in contrast the majority (n = 184; 66.7%) did not agree that there were negative effects.

As shown in Table 2, more than half the respondents (overall n = 142; 51.4%; proportionally more nonprincipals than principals) agreed that appraisal had an impact on CPD, and encouragement of their CPD through appraisal was reported by 55.8% (overall n = 154; 55.8%; proportionally more non-principals than principals) whilst fewer (n = 56; 20.3%) disagreed. This finding was reinforced by the majority of respondents (n = 196; 71.0%) who disagreed that appraisal had discouraged CPD.

Five themes emerged from the analysis of responses to open questions which asked respondents to explain how the appraisal process had impacted on learning, practice and CPD.

Many respondents reported that the appraisal process had been positive and challenging and had encouraged them to focus on their learning needs; some identified that without the appraisal process they would have spent little time on CPD. The process for some encouraged prioritisation of learning, improved confidence in practice and focused learning on practice needs.

‘Gives a focus to work towards, more of an incentive to professional education’ (Respondent 259)

‘Increased confidence’ (Respondent 91)

‘Has motivated me to direct my learning to needs within the practice’ (Respondent 116)

‘Much more home learning, much better organised’ (Respondent 175)

Documentation

More organisation and documentation of learning emerged; many considered designing a PDP with objectives as beneficial, whilst others identified the additional paperwork as an exercise which did not enhance the learning process.

‘Have become more methodical and have a system to access information needed’ (Respondent 30)

‘I know I have to log everything from an educational point of view, it is irritating to collect all this information, who is going to look at it? (Respondent 203)

‘Writing a PDP helps to plan realistic aims for the coming year’ (Respondent 276)

Reflection

The appraisal process encouraged respondents to reflect on both their learning needs and their practice. Reflection identified areas of weakness which could be remedied, and allowed individuals to consider different methods of learning and review of current systems.

‘Reflecting on my practice has highlighted areas of weakness I can address’ (Respondent 209)

‘It has made me think more about what kind of learning I respond best to, and allows me not to go to events which I know I will not enjoy’ (Respondent 49)

‘Appraisal does encourage reflection; it does encourage you to look at systems in place’ (Respondent 47)

Some respondents identified that they already followed the principles of adult learning and that the appraisal process was not required to maintain good practice. The appraisal process had not changed existing learning habits but reinforced them and consolidated the role of audit and significant event analysis within practice. It was identified that appraisal created opportunities for discussions with a respected peer and that appraisers provided constructive challenges to existing attitudes, were helpful in responding to new ideas and could suggest alternative approaches to professional development.

‘I have always given great importance to continuing education and appraisal has not changed this’ (Respondent 193)

‘I feel it has largely validated what I already do’ (Respondent 122)

‘I have effected changes identified by the appraiser, significant event analysis and more audit is now done’ (Respondent 169)

‘Recently my appraiser has challenged me and made me look at alternative ways to develop professionally’ (Respondent 131)

External drivers, productivity and time issues in appraisal

Some respondents considered that the appraisal process was driven by external sources, and was a mechanistic exercise, time-consuming and non-productive waste of time.

‘I see appraisal as a politically motivated exercise which wastes time, costs public money, diverts attention from patient care and generates stress’ (Respondent 37)

‘I am now ticking boxes instead of doing self-directed learning’ (Respondent 180)

‘This has been an extra time imposition without any change in my practice; I do it because the GMC expect me to do it’ (Respondent 34)

Based on their experience of the appraisal system, respondents were asked for any recommendations or ideas for change which they felt would be relevant and helpful. Comments were received from 170 (61.6%) respondents and five core themes were identified; purpose of appraisal and links with revalidation, the appraisal process as an educational tool, the role of the appraiser, the effect of the appraisal process on time and the use of resources.

Purpose of appraisal and the links with revalidation

Respondents identified that the main purpose of appraisal and the exact role it will play with revalidation was a cause for concern and an area that needs clarification.

‘A firm need for appraisal to remain in its present style, a process towards revalidation is not appropriate’ (Respondent 262)

‘Far too long waiting for appraisal process to evolve into something meaningful, still do not know how appraisal and revalidation will link’ (Respondent 88)

It was considered by many that the appraisal process should be kept as a formative and confidential tool, while others felt there may be advantages in making the appraisal process summative.

‘I feel that its facilitative and formative nature is valuable and helpful’ (Respondent 85)

‘Keep it confidential, time to discuss worries is essential. Should not be pass or fail. It should include a PDP’ (Respondent 13)

‘Needs to be summative rather than formative to provide sufficient motivation to attain goals’ (Respondent 81)

The role of the appraiser was debated; while some valued the supportive relationship from a peer other respondents offered different models that could be implemented.

‘The most important part of appraisal process, is to be able to mull over thoughts with a trusted colleague’ (Respondent 255)

‘I think local GPs appraising colleagues is counterproductive’ (Respondent 5)

‘Currently the process has no penalties, I think it should be an unknown appraiser and there should be some gain or loss according to outcome’ (Respondent 71)

It was identified that the appraisal process and education are both time-consuming, with extensive paperwork (which was seen as a burden), and many suggested that the allocation of protected time would be appreciated.

‘Need to recognise that time for learning is needed within the working week’ (Respondent 2)

‘The onerous paperwork should be stopped, let us say what interests and drives us in our own words’ (Respondent 73)

Some respondents suggested that the current system should be stopped altogether as it was not a good use of time and financial resources. A range of alternative suggestions included use of e-portfolios, audit of performance, introduction of five-yearly skills assessments and the use of a formal examination system.

‘Scrap it; it is a complete waste of time and money’ (Respondent 52)

‘Scrap it, introduce an e-portfolio that requires entries for educational meetings and reflective work, but is not onerous’ (Respondent 63)

‘Make appraisal more task oriented, use audit to measure performance and change’ (Respondent 272)

‘Introduce an exam system to test knowledge and diagnostic awareness and consulting skills’ (Respondent 270)

‘Why not have five-yearly clinical skills assessments for those who want it’ (Respondent 12)

Discussion

Findings of this survey offer insights into the appraisal experiences of GPs and the impact of appraisal on their learning, practice and CPD. For the majority of respondents, the appraisal process enhanced learning, improved practice and encouraged CPD. This was supported by the qualitative findings which highlighted that the appraisal process had been an encouragement, facilitating a focus on learning needs, increasing confidence in practice, providing an incentive for CPD and providing benefits of personal development planning in the form of a written PDP. Overall, the findings of this study demonstrate that the appraisal process enhanced learning, improved practice and encouraged CPD for the majority of respondents.

Strengths of this study were the combination of approaches utilised in questionnaire design, qualitative responses enhancing explanation of the quantitative ratings, and the response rate. The final overall response rate (71%) achieved in this study can be regarded as good compared with Barclay et al[14] who achieved a final response rate of 67% in a postal questionnaire survey of GPs following secondary mailings. Reasons for the relatively good response rate may have been because the appraisal process is compulsory for all doctors and the subject is topical, highly relevant to everyday practice and elicits strong views.[15]

The study was a regional survey and findings may not be representative of other regions in England. The sample may also be under-representative of women GPs in Kent. A limitation of the study was that most responses were obtained from practice principals and potential bias cannot be excluded as non-responders (29%) may have held different views. Breakdown of response rates revealed that the response rate for practice principals was 81% (256/315) and that for non-principals was 28% (20/70). The findings may have been more representative of practice principals. Use of inferential statistics was precluded by the small sample size in some respondent categories

Earlier studies have shown that the implementation of appraisal in general practice settings can be challenging and have identified a number of barriers to and facilitators for success.[16,17] Our findings support those of Middlemass and Siriwardena who demonstrated that doctors welcome appraisal provided it has local ownership, an educational purpose and results in an agreed PDP.[18] They concluded that GPs who felt more control over the process were more positive towards it.

Other positive findings in this study were that the current process led to enhancement of reflection and increased use of audit and significant event analysis, which highlighted areas of weakness needing attention and provided insights into preferred individual learning styles. These findings concur with those of Cross and White who found that a variety of educational tools were valued as aids to learning and significant event auditing in particular was regarded as favourable. [19]

In contrast, fewer respondents revealed negative attitudes and experiences. Notable amongst these was that the appraisal process did not enhance learning, practice and CPD. Furthermore, some regarded it as a waste of public money, not required to maintain good practice and simply providing a source of documentation for others. Similar negative findings arose from the study by Cross and White who found that while professional development was considered part of a career pathway to keep GPs up to date over 50% of GPs regarded the completion of a PDP following appraisal as a ‘hoop jumping’ exercise.[19]

The qualitative findings enabled respondents to express their opinions and voice their ideas for future change. The vital role of the appraiser as a respected peer withwhomconstructive discussionswerewelcomed emerged as an important finding. However, other respondents stated that in future appraisers should not be drawn from the local GP population, but be independently appointed. These views concur with those found in a study by McKinstry et al who compared the experience of being appraised by either a GP partner or an external peer.[20] They identified that collusion between appraiser and appraisee and a lack of local knowledge could be disadvantages. Woods et al found that the choice and skills of the appraiser have an important bearing on the process.[21]

A common view expressed was that the appraisal process is time-consuming, with no protected time for CPD or documentation and the preparation required for the appraisal. Boudioni et al found that provision of protected time was a facilitator for lifelong learning whereas time constraints and workload pressures were a barrier to learning and development.[16] Middlemass and Siriwardena also highlighted that the two main concerns about the appraisal process were the time involved and the lack of resources for the process.[18] Extensive documentation and goal setting emanating from appraisal and personal development planning was not always considered necessary by respondents, as many considered that they were adult learners who were already responsible for addressing their own learning needs. This positive attitude for accountability towards learning supports the vision that lifelong learning is central to the delivery of patient-centred care.[22]

An interesting recommendation was that the current documentation should be replaced by an e-portfolio. This tool is now used for general practice training, and many trainers are now familiar with its use. The suggestion is that the e-portfolio could be used in the appraisal process as an electronic assessment tool, as it may provide learners and trainers with evidence of progress that can be interrogated at local and national levels to ensure consistent high-quality training (www.e-lfh.org.uk).[23] The role of appraisal as a formative or summative tool and the exact role it will play in revalidation was a commonly identified topic and a cause for concern, as highlighted by McKinstry et al.20 Appraisal will undoubtedly take on a summative element as it links with revalidation; this is an area of concern identified in this study, and also highlighted by Jennings.[24]

Implications for future practice and research

This study was conducted in June 2008, prior to the publication of Medical Revalidation: principles and next steps8 which has clarified many of the issues highlighted by respondents in this study. Variability in the current appraisal processes is recognised together with the need for a new system to be introduced which is accepted and valued by doctors. The future approach to annual appraisal will undoubtedly take on a summative element, as appraisal is described as the key vehicle confirming that a doctor is maintaining satisfactory fitness to practise and that issues of concern are being managed effectively.8 The new approach to annual appraisal as a basis for revalidation will provide opportunities to remedy potential issues of concern at an early stage and to provide a predominantly formative system for doctors. Only time will tell whether this new approach to appraisal and revalidation will address the current concerns of GPs.

Conclusion

This study has provided an opportunity to explore GPs experiences of the appraisal process. The current appraisal system should reflect the diversity of medical practice but a standardised model based on theGMC’s Good Medical Practice should be developed for revalidation. 9 As the role of appraisal within the revalida tion process changes it is recognised that the quality, consistency and nature of appraisal will be essential to ensure the confidence of patients and doctors.

References

1. Department of Health White Paper. Trust, Assurance and Safety: the regulation of health professionals. Department of Health: London, 2007.
2. Chief Medical Officer. Good Doctors, Safer Patients. Department of Health: London, 2006.
3. Department of Health.Supporting Doctors, Protecting Patients. The Stationery Office: London, 1999.
4. Department of Health.The NHS Plan. The Stationery Office: London, 2000.
5. Mohanna K. Appraisal: what now? Education for Primary Care 2003;4:535–6. 6 General Medical Council.Revalidating Doctors. GMC: London, 2000.
6. Smith J. The Shipman Inquiry, fifth report; safeguarding patients: lessons from the past – proposals for the future. The Stationery Office: London, 2004.
7. Chief Medical Officer for England’s Working Group, Department of Health. Medical Revalidation: principles and next steps. Department of Health: London, 2008.
8. General Medical Council.Good Medical Practice. www. Gmcuk.org/guidance/good_medical_practice.asp
9. Charlton R. Personal development plans. British Medical Journal 2002;325:Career Focus s36–s37.
10. Adams R, Illing J, Jelley D, Walker C and van Zwanenberg T. The Critical Success Factors in Delivering GP Appraisal. Sunderland: Postgraduate Institute for Medicine and Dentistry, 2005.
11. Bell J. Doing your Research Project. Maidenhead: Open University Press, 2006.
12. Hays R. Research Degrees for Health Professionals. Oxford: Radcliffe, 2007.
13. Barclay S, Todd C, Finlay I, Grande G and Wyatt P. Not another questionnaire! Maximizing the response rate, predicting non-response and assessing non-response bias in postal questionnaire studies of GPs. Family Practice 2002;19:105–11.
14. Edwards P, Roberts I, ClarkeMet al. Increasing response rates to postal questionnaires: systematic review. British Medical Journal 2002;324:1183.
15. Boudioni M, McLaren S, Woods LP and Lemma F. Lifelong learning, its facilitators and barriers in primary care settings: a qualitative study. Primary Health Care Research and Development 2007;8:157–69.
16. McLaren S, Woods LP, Boudioni M, Lemma F and Tavabie A. Implementing a strategy to promote lifelong learning in the primary care workforce: an evaluation of leadership roles, change management approaches, interim challenges and achievements. Quality in Primary Care 2008;16:147–5.
17. Middlemass J and Siriwardena AN. GPs, revalidation and appraisal: a cross sectional survey of attitudes, beliefs and concerns in Lincolnshire. Medical Education 2003;37:778–85.
18. Cross M and White P. PDPs, the Wessex experience. Education for Primary Care 2004;15:205–12.
19. McKinstry B, Peacock H and Shaw J. GP experiences of partner and external peer appraisal. British Journal of General Practice 2005;55:539–43.
20. Woods L, Boudioni M, Mclaren S and Lemma F. Delivering the quality agenda: the experience of implementing appraisal systems in primary care. Quality in Primary Care 2006;14:21–7.
21. Department of Health. Working Together – Learning Together. The Stationery Office: London, 2001.
22. E-learning for Healthcare. www.e-lfh.org.uk Department of Health, 2009.
23. Jennings S. PDPs and self-directed learning for healthcare professionals: are they evidence based? Postgraduate Medical Journal 2007;83:518–24