Research Paper - (2004) Volume 12, Issue 2
Honorary Senior Research Fellow
Roy Powell PhD BSc (Hons) MIBIOL CBIOL ILTM
Coordinator/Honorary Research Fellow
Judy Jolliffe MSc GradDipPhys
Project Leader
Kieran Sweeney MA MPhil FRCGP ILTM
Clinical Fellow
Katrina Wyatt PhD BSc (Hons)
Lecturer
Exeter and North Devon NHS Research and Development Support Unit, Peninsula Medical School, Universities of Exeter and Plymouth, UK
Received date: 06 November 2003; Accepted date: 18 November 2003
Context Non-executive directors in the South West Region.Objectives To evaluate the effectiveness of a one day training programme for non-executive directors in critically appraising board clinical governance reports.Design The evaluation included delegate perceptions of their skills, knowledge and confidence, as well as the views of their Board chief executive officers (CEOs)/chairs regarding the ability of the non-executive directors to bring these attributes to board meetings, before and after the workshop.Subjects In total, 86 non-executive directors attended the workshops. All eight healthcare communities in the south west were represented and most participants were drawn from primary care,health authorities, and acute and community trusts.Results Results showed that delegates’ knowledge,skills, attitudes and confidence had improved significantlyimmediately after the workshop. In particular,delegates’ knowledge and understanding of the critical appraisal aspects of clinical governance seemed to have been the most affected.The evaluation strategy also tested whether these skills and knowledge were retained after three months. Results showed that the gains made by the workshop had been sustained in all cases. In some areas, such as skills and confidence in their experience as working as non-executive/lay representatives,their ratings of performance had continued to improve beyond the experience ofthe workshop.Findings also showed that CEOs/chairs noticed an increase in delegates’ confidence in carrying out their tasks as non-executive/lay members. Ratings of their contribution to the board meetings improved but not significantly.Conclusions Overall, the workshop was shown to be an effective mechanism for raising the performance of non-executive/lay members in being fit for the role for critically appraising clinical governance reports.
clinical governance, critical appraisal, evaluation, non-executive directors, training programme
The NHS Executive guidance on clinical governance states ‘that organisations should ensure that they have appropriate mechanisms in place to deliver routine board reports on progress made implementing clinical governance’.[1] It is, therefore, essential that all directors are familiar with the concept of clinical governance and are able to understand and interpret the different types of information being presented to them.
Lilley states that clinical governance is about being open about the strengths and weaknesses of what we do; being determined to improve by adopting and sharing the best practice we can find; making our own contribution more valuable through continuous personal development, comparing ourselves with the best; and by listening to the people we serve.[2]
Non-executive directors have a key role to play in influencing the implementation and monitoring of clinical governance arrangements to ensure clinical excellence in their healthcare organisation.
In 2000, the Secretary of State appointed 138 chairs and 660 non-executive directors to NHS trusts and health authorities and around 50 chairs and 250 nonexecutive directors to primary care trusts (PCTs).[3] The qualities required of non-executive directors should include:
• having a strong personal commitment to the NHS
• good communication skills
• commitment to the public service values of accountability, probity and openness
• plenty of common sense
• possible experience as a carer or user of the NHS
• possible previous local government or local community service.[3]
This paper reports on the implementation and evaluation of a one-day training programme to provide non-executive directors with the opportunity to develop their confidence, knowledge and skills in critically appraising board clinical governance/quality reports.
The training programme was provided by the Exeter and North Devon NHS Research and Development Support Unit (University of Exeter) and was overseen by an NHS Executive (NHSE) regional advisory group consisting of the regional R&D (research and development) Manager, the regional Assistant Director of Public Health and chaired by a non-executive director of an acute trust board.[4,5]
The one-day training programme was provided in each of the eight healthcare communities within the South West region during 2001 over a three-month period.[6] The content of the programme was developed following a survey of the needs of 129 non-executive directors (response rate = 44%) and follow-up telephone discussions.[7] In response to the statement, ‘I am quite clear about the nature of clinical governance and what it means’, 68.9% of respondents agreed or strongly agreed. However, in response to the statement, ‘I find it difficult to appraise clinical governance reports’, 47.9% agreed or strongly agreed.
Table 1 shows the relative importance of items identified in the needs assessment exercise.
Although there were variations between different groups of non-executive directors (i.e. health authority, primary/secondary care trusts) in their expressed needs for training, Table 1 shows that there were similar rankings for the top ten needs. These included training in poor performance, risk assessment, adverse events, where to find the evidence, clinical audit, significant event reporting, accountability, involving users/carers and no blame reporting.[8]
Learning outcomes were to:
• raise awareness of clinical governance issues
• critically appraise clinical governance reports
• identify strategies for addressing clinical governance issues within board meetings
• put clinical governance in the context of theNHS as a whole.
The programme is described in Table 2.
Sample
Of the non-executives approached, 86 attended the workshops. A breakdown of participants by location and organisation is given in Tables 3 and 4.
Table 3 shows that the workshops involved all the keyNHS healthcare organisations and that most of the participants were drawn from primary care, health authorities, and acute and community trusts. Table 4 shows that all healthcare communities were representedwith most of the workshop participants coming from Dorset, South and West Devon, and Somerset areas.
Evaluation procedure
Both process and outcome methods were adopted in the evaluation.
At the conclusion of each workshop, delegates were asked to complete a workshop evaluation form. This form examined issues around the content and relevance of the pre-reading, the short presentations by the facilitators, the plenary sessions and their enjoyment of the workshop, as well as organisational issues such as venue and catering. Also delegates were asked for their suggestions about how the workshop could be improved.
The outcome evaluation strategy was twofold and end-user focused. The first major aspect of the evaluation involved delegates’ perceptions of their attitudes, knowledge, skills and confidence in being a non-executive director or lay representative. Delegates were asked to complete an evaluative questionnaire at three stages: (1) prior to the workshop, (2) immediately after the workshop, (3) follow-up three months later. The follow-up questionnaire was posted to them. This evaluation was similar to the instrument developed by Taylor et al for evaluating Critical Appraisal Skill Programmes (CASPs).[9] Analyses were conducted using repeated measures and non-parametric statistics where necessary using SPSS version 9.0.
The attitudes of delegates were examined using nine items that were marked using a Likert scale from ‘strongly agree’ to ‘strongly disagree’. Knowledge was tested using a 16-item scale developed around clinical governance terms and included concepts such as critical appraisal, literature searching, continuing professional development and quality improvement. Skills were tested using an eight-item scale and examined issues such as the contribution of nonexecutive directors to board discussions and their ability to interpret reports. Confidence was assessed in two ways. Part A included a 17-item scale, which measured delegates’ confidence in terms of their experience of working as a non-executive director or lay representative (participation confidence). This aspect was not tested immediately after the workshop because delegates would not have had the opportunity to experience another board meeting until after the workshop. Part B consisted of a seven-item scale, which described the confidence in their own ability to carry out certain tasks or skills (task confidence).
The second major aspect of the evaluation involved the chief executive officers (CEOs) or chairs of boards. They were posted an evaluation questionnaire before and three months after the training programme about their views of the effective participation of nonexecutive directors. The questionnaire focused on two aspects: first, how they would rate the confidence of their non-executive/lay member(s) in carrying out certain tasks (seven-item scale). This scale was identical to the one completed by delegates on ‘task confidence’; secondly, how they would rate the contribution of such members at board meetings (1–10 sliding scale). Analyses were conducted using non-parametric statistics where appropriate (i.e. Friedman’s test and Wilcoxon matched-pairs signed-ranks test).
CEOs were sent a questionnaire only if the chair attended the training programme. If the workshop was effective, then the chairs/CEOs should notice improved confidence in their non-executive/lay members and also experience a qualitatively different interaction with them at subsequent board meetings.
The findings are discussed in terms of process and outcome evaluation.
A total of 79 delegates (92% response rate) completed a workshop evaluation form. A summary of the results is provided in Box 1.
Box 1 shows that, overall, the evaluations were very positive. Most delegates found the content, venue and catering to be of a high standard and relevant to their needs as non-executive/lay representatives.
All 86 delegates completed a pre-evaluation form. Of these, 82 completed the post-evaluation form (95% response rate). Four were lost due to delegates departing the workshop slightly early; 67 delegates completed the three-month follow-up evaluation form (82% response rate).
The results are presented in Table 5, and Figure 1 focuses on the key evaluation areas: knowledge, skills, attitudes and confidence. Therefore data were analysed using a repeated measures analysis. Data for the aggregate scales of the key evaluation areas were not all normally distributed and were summarised using median percentage scores and interquartile ranges (IQRs). They were analysed using Friedman’s test for repeatedmeasures or in the case of ‘participation confidence’, Wilcoxon’s matched pairs signed ranks test.
Figure 1: Percentage scores for delegates pre- and post-workshop and at three months’ follow-up
The aggregate knowledge scale consisted of 16 items. Median scores are given in Table 5 and Figure 1 provides a graph of the overall results. Results showed that there was a significant improvement (P = 0.001) in delegates’ knowledge of critical appraisal terms immediately following the workshop. That is, there was an increase in understanding of terms and how to define them. There was a large increase in score from 70% to 85.6% immediately after the workshop, and the score remained well above pre-workshop levels three months later. These results provide evidence that the significant impact of the workshop on delegates’ knowledge was sustained over time.
An examination of individual items within the knowledge scale showed that awareness and understanding of delegates had been significantly improved and sustained for 13/16 terms as follows:
• significant event audit
• data type
• critical appraisal
• evidence-based medicine
• standards
• continuing professional development
• accreditation
• Medline
• rating scale
• quantitative vs qualitative analysis
• capacity and capability issues
• risk management
• systematic review.
Understanding of the term Cochrane Collaboration significantly improved (P < 0.0001) after the workshop but declined after three months. However, the follow-up score remained significantly higher (P < 0.0001) than the pre-workshop score.
The only knowledge term that did not improve as a result of the workshop was quality improvement. One reason for this outcome is that the workshop did not explicitly discuss the term, hence, it was not surprising that delegates did not improve in this area of understanding.
Skills
The aggregate skill scale consisted of eight items. As Figure 1 shows, skills were the lowest rated of the five areas investigated, scoring only a median of 52.5% pre-workshop. Results showed that there was a significant improvement (P < 0.001) in delegates’ skills in discussing and appraising clinical governance reports after the workshop (i.e. post-test) and at the threemonth follow-up stage (see Table 5). These results provide evidence that the significant impact of the workshop on delegates’ knowledge was not only sustained, but also accelerated over time.
An examination of individual items within the skills scale showed that some skills improved immediately after the workshop while other skills developed three months after the workshop. For example, delegates perceived that the workshop had an impact on their skills of being able to contribute fully to the discussion and being able to get hold of information that would support their arguments. Both these skills were sustained over time. However, their skills in areas such as understanding research evidence and statistics in papers, accessing user views, interpreting reports and understanding jargon at meetings, which would not have been expected to immediately improve, were indeed found to have improved three months after the workshop. The only skill that did not improve at any stage was finding the time to prepare for meetings.
Attitudes
The aggregate attitude scale consisted of nine items. Figure 1 shows that attitudes were the highest rated of the five areas measured before the workshops were attended with amedian score of 88.9% (also see Table 5). Results showed that there was a significant improvement (P = 0.001) in delegates’ attitudes and commitment towards clinical governance after the workshop. However, this impact was not sustained after three months.Onepossible reasonfor thiswas the ceiling effect, where scores for this aspect of the evaluation were already high. The pre-workshop median score was 88.9% and increased to 91.1% at the end of the workshop. The three-month follow-up median score fell back to 88.9%, which was not significantly different from the pre-test score (P = 0.864).
The aggregate task confidence scale consisted of seven items. Results showed that, after the workshop, there was a significant improvement (P < 0.001) in delegates’ confidence in carrying out their tasks as nonexecutive/ lay representatives. This improvement was sustained after three months. The pre-workshop median score was 77.1% and increased to 81.9% at the end of the workshop. The three-month follow-up median score was 82.9%.
An examination of individual items within this task confidence scale showed that the workshop impacted more strongly on delegates’ confidence in the following areas:
• effectively conveying the views of other users
• asking for clarification
• commenting on guidelines/protocols
• challenging health professionals when necessary
• helping with surveys/research.
While confidence in presenting evidence and keeping attention focused on important issues was improved as a result of the workshop, both scores in these areas were reduced slightly after three months.
Delegates’ participation in board meetings as non-executive/lay representatives
The aggregate participation confidence scale consisted of 17 items. Wilcoxon’s matched-pairs signed-ranks test was used to compare the differences between pretest and three-month follow-up median scores. Results showed that delegates’ experiences of working as a non-executive/lay representative were significantly improved (P = 0.001) three months after the workshop. Although the pre-workshop median score of 60% was the same as the three-month followup median, the IQR had shifted upwards (see Figure 1).
An examination of individual items within this participation confidence scale showed that the areas that were most affected by the workshop were:
• putting one’s views across
• feeling that their advice is sought
• feeling that their ideas are sought
• feeling respected
• feeling an equal member of the team
• feeling that they have an influence on decisions
• feeling that they contribute well to their organisation.
The following areas showed no significant improvement (P > 0.05):
• feeling patronised
• feeling under pressure to take sides
• believing that they are knowledgeable
• enjoying the meetings
• learning a lot about the topics discussed at meetings
• feelingthat they cansayopenlywhat theythinkabout the strengths and weaknesses of their organisation.
Figure 1 provides a summary of the impact of the workshop on knowledge, skills, attitudes and confidence. The graph indicates that the immediate impact of the workshop was felt on the knowledge gained by the delegates. This is not surprising, given that knowledge is relatively easier to improve than areas such as skills, attitudes and confidence. However, it appears that the workshop had a lingering effect on delegates’ skills and confidence since these were the only areas whose follow-up scores were significantly better than posttest scores. This result is reassuring in that it suggests the workshop provided delegates with the necessary information to allow them to further develop their role as non-executive/lay representatives.
Pre-evaluation forms were sent to the chairs or CEOs of the corresponding NHS trust or primary care group (PCG) board. A total of 48 were returned (56% response rate) and 45 follow-up evaluation forms were returned (94% response rate).
The aggregate task confidence scale consisted of seven items. Figure 2 provides a graph of the overall results. Ratings by CEOs/chairs of delegates’ confidence in carrying out certain tasks improved but not significantly (P = 0.069) three months after the workshop. The pre-workshop median score was 77.1% and the three-month follow-up median score was 80%. Had the response rate been greater, it is possible that the difference may have proven significant.
Figure 2: Comparison of task confidence rated by delegates and CEOs/chairs
An examination of individual items within the task confidence scale showed that the areas in which CEOs/ chairs felt most improvement were:
• effectively conveying the views of other users
• commenting on guidelines/protocols.
CEOs/chairs did not find significant improvements (P > 0.05) in the following areas:
• helping with surveys/research
• asking for clarification
• presenting evidence
• keeping attention focused on important issues
• challenging health professionals when necessary.
Interestingly, a comparison between delegate and CEO/chair ratings of confidence was found to be very similar (see Figure 2), adding some validity to these findings.
CEOs/chairs rated the contribution of non-executive/ lay representatives at board meetings using a scale of 1–10, where 1 = little contribution and 10 = high contribution. No significant difference (P = 0.439) in scores was found between testing periods. The median score at pre-test was eight and at three-month followup it was seven.
The evaluation findings, based on the attending 86 non-executive/lay representatives, have shown that the workshop was received favourably by its participants. Most found the content of the workshop, venue and catering to be of a high standard and relevant to their professional development.
Delegates’ perceptions of the impact of the workshop on their knowledge, skills, attitudes and confidence were also recorded. Overall, results showed that all these areas had improved significantly immediately after the workshop. In particular, delegates’ knowledge and understanding of the critical appraisal aspects of clinical governance seemed to have been the most affected.
The evaluation strategy also gauged whether these skills and knowledge were retained after three months. Results showed that the gains made by the workshop had been sustained in all cases. In some areas, such as skills and confidence in their experience as working as non-executive/lay representatives, their ratings of performance had continued to improve beyond the experience of the workshop.
Another feature of the evaluation strategy was to seek the views of CEOs/chairs of health authorities and trusts about the participation of delegates at board meetings. Findings showed that CEOs/chairs had noticed an increase in delegates’ confidence in carrying out their tasks as non-executive/lay members. Ratings of contribution to the board meetings improved but not significantly.
Many of the skills, knowledge, attitudes and confidences of the delegates seemed rather high scoring before the implementation of the workshop. It appears that those who attended the workshops were most likely to be already skilled in appraising and implementing clinical governance reports. One wonders whether those non-executive/lay representatives who did not attend the workshop may have shown more significant improvements in the targeted areas. Despite the possibility of a well-informed audience and hence the ceiling effect, the workshop has been shown to be an effective mechanism for raising the performance of non-executive/lay members in being fit for the role for critically appraising clinical governance reports.
None
