Research Paper - (2010) Volume 18, Issue 1
1Professor of Health Innovation, Curtin Health Innovation Research Institute, Curtin University of Technology, Perth, Western Australia, Australia
2Professor in General Practice, PHC RED Senior Research Fellow, Discipline of General Practice, University of Queensland School of Medicine, Brisbane, Queensland, Australia
3Associate Professor School of Medicine and Public Health, University of Newcastle, New South Wales, Australia
4Professor, Director, Centre for Health Research and Psycho-oncology (CHeRP), Cancer Council New South Wales, University of Newcastle and Hunter Medical Research Institute, Newcastle, New South Wales, Australia
5Research Fellow, Discipline of General Practice, School of Medicine, University of Queensland, Brisbane, Queensland, Australia
Received date: 18 September 2009; Accepted date: 24 November 2009
BackgroundThis study aimed to develop an innovation to assist general practitioners (GPs) in Australia to proactively address the needs of caregivers of people with cancer. MethodSix GPs were video recorded each consulting six actor-patients in their respective practices. All cases depicted caregivers of people with cancer. The patients were instructed to complete a Needs Assessment Tool for Caregivers (NAT-C), before the consultation. Actor-patients were instructed to present the NAT-C to three of the six GPs they consulted, selected at random. Two assessors independently reviewed each consultation performance using the Leicester Assessment Package (LAP). The practitioners and actor-patients focused on the value of the NAT-C and how it could be deployed to best effect in a subsequent ‘stimulated recall session’. ResultsThirty-four consultations were successfully recorded. The mean duration of consultations was 13 min. 47 sec. (range 6 min. 3 sec. to 22 min. 51 sec.). GPs differed in core competencies as measured by the LAP (P0.001), range 37–92%. However, they demonstrated no significant differences in performance (LAP scores) analysed by scenario (P = 0.99). The ‘generalised estimating equation’ (GEE) model identified an improved LAP score in consultations in which the NAT-C was used (average of 3.3 points; 95% CI: –3.99, 10.6), after controlling for the different GPs and scenarios, but this improvement was not statistically significant (P = 0.37). The participants felt that the NAT-C was beneficial and suggested how it could be further refined. ConclusionsIf this innovation had been formally tested in a randomised trial without assessing its impact on the consultation there might have been significant difficulties with administering the intervention in practice.
cancer, caregivers, consultation, general practice, innovation.
How this fits with quality in primary care
What do we know?
Many cancer sufferers are supported in the community by relatives who are carers. Little is known about strategies for assessing needs of carers for patients with cancer. A new instrument for prior completion by carers to aid the consultation, the NAT-C, was developed to help improve the quality of consultations for caregivers.
What does this paper add?
This study aimed to assess the potential impact on the quality of the consultation of introducing the NAT-C to help GPs proactively address the needs of caregivers of people with cancer by using standardised assessment of audio or video recordings of simulated consultations incorporating the instrument. Although there was no objective improvement participants felt that the NAT-C was beneficial and suggested how it could be further refined.
One in four Australians will develop cancer before the age of 75 years.[1] Most patients will be older adults and therefore vulnerable to the impact of demanding surgery and complex regimens of adjuvant therapies.[1,2] The trajectory through the illness will be punctuated by appointments at specialist clinics and, in many cases, admissions to hospital. The majority of people with cancer will survive.[1] Most will be supported during their treatment by a partner or other close family member or members, who are sometimes equally frail or vulnerable.[3] Some caregivers may be ambivalent about assuming the supportive role, and others will delay seeking support for their own needs, choosing instead to invest their time and energy on the needs of the person with cancer.[4] Caregivers may feel unworthy of support or confused about their role: they may regard offers of help as a mark of weakness or as a betrayal of their duty to the person with cancer.[5–7] Consequently some caregivers are at risk of complex, preventable and treatable problems, including mental and physical illness. There are many funded programmes in Australia that could support caregivers of cancer patients, including the Cancer Helpline and the Commonwealth Carer Resource Centre. Caregivers may not be aware of these services. In many instances access to specific community support services/service providers necessitates referral by a GP. Therefore caregivers need to be offered the opportunity to reflect on whether to enlist this support. Their own GP is a logical person to review their needs and facilitate appropriate referrals.
Annually there are 100 million consultations with GPs in Australia.[8] Each encounter is an opportunity to proactively assess the needs of the patient. Since most consultations will last no more than 15 minutes it is important that problems are presented efficiently and in a way that allows the doctor to recognise and negotiate a response to need.[8]
The rationale for focusing on the consultation in this study is framed within two theories relating to meeting needs in a professional setting. Jean Watson’s theory of caring originally referred to nursing but can be applied to medical practice.[9–11] Viewing the person’s health through Watson’s caring lens, a critical element is the quality of the interaction between the patient and the caregiver, in this case the GP. Also in general practice, patients may present with concerns in many parts of their lives and the GP must address each effectively. Rubinstein et al describe two processes that help people switch between tasks unconsciously.[12] One is ‘goal shifting’ (‘I want to do this now instead of that’), the other is ‘rule activation’ (‘I’m turning off the rules for that and turning on the rules for this’). Problems arise when switching compromises productivity and safety, both of which are required in general practice. Thus, diagnostic and therapeutic errors may occur when either process is compromised.
The impact of an innovation that requires practitioners to attend to multiple issues in the consultation needs to be assessed, and can be measured indirectly with reference to measures of core competencies in general practice. It is recognised that consultation performance is a product of competence, the influences of the individual (e.g. health, relationships) and the influences of the system (e.g. facilities, practice time). Therefore the most appropriate context in which to observe the impact of innovations in a healthcare system is within the practitioners’ own premises and subject to the constraints of that setting.[13] The aim of this study was to develop an innovation to be tested in a formal clinical trial in Australian general practice. We conceived of this study as a vital Phase I study in developing a complex intervention.[14] We aimed to road test the intervention and assess its likely impact on the quality of the consultation.
Setting
The study involved video or audio recorded consultations with GPs in which actors were portraying the caregivers of people with cancer. These sessions were conducted on the premises of the participating GPs in Perth, Western Australia. Video assessment of GPs in daily practice is a valid and reliable method. There is a trade-off between feasibility on one hand and validity, reliability and credibility on the other hand. Compared to investments in observation methods in standardised settings, the costs of video observation of GPs’ actual performance are acceptable.[15]
Actor-patients
All six ‘patients’ presented themselves as caregivers of people with cancer. The scenarios were developed by the members of the team and are outlined in Box 1. Physical signs, presented as descriptions, were available if the GP proposed relevant physical examination. Therefore no actor was subjected to a physical examination during the study. Patients were amateur actors trained to portray the relevant case. A brief medical record with the relevant past medical history was prepared for each patient and was available to the GP at his or her surgery. The patients presented to the doctors as routine appointments but because of the need for high quality recording the consultations were identifiable as study cases by the GPs.
Innovation
A needs assessment tool for caregivers (the NAT-C) was developed by the project team. The commonly reported needs of lay caregivers of cancer patients were identified from the literature. These needs were transformed into simple statements and grouped into several key domains of caregiver health and wellbeing: information issues, practical issues, personal health and wellbeing issues, relationship issues and meaning issues. An assessment form was formatted, based on these domains, and additional space was left at the end of the assessment for caregivers to note any other topics of concern. An explanatory cover sheet was prepared to indicate to caregivers and their GPs the purpose of the NAT-C, and how to use it. Caregivers were asked to rank the level of concern which each statement represented to them, by selecting one of three options: ‘None’, ‘Some’ or ‘A lot’. They were then asked to identify the items of concern which they wished to discuss with their GP, either now or later. If there were several issues to discuss, caregivers were asked to prioritise the top three. Five GPs and 20 lay people associated with the academic team informally reviewed the draft NAT-C prior to administration, with a focus on layout, language and functionality.
Consultations
Six GP volunteers were asked to consult with the actor patients as though the person had previously visited the practice for one or two ongoing medical problems. The practitioners were allocated 15 minutes per consultation; this is the standard time allocated to GP consultations in Australia. The scenarios were presented to the practitioners as consecutive cases. GPs were asked to make clinical notes and outline any management plan in as much detail as they would in their practice. GPs were informed that the study was about developing an innovation to assist caregivers. The methods outlined here replicate similar studies to develop innovations or test the impact of multitasking in GP consultations in Australia.[16,17] An example of a consultation has also been presented in an online journal.[18] For the purposes of this study actor-patients were instructed to present the NAT-C at all consultations with three of the GPs selected at random.
Outcome measures
Quality of consultation
The LAP has been shown to facilitate reliable assessments of consultation performance and its face validity has been confirmed for general practice consultations. [19,20] Three of the six LAP categories of consultation competence (interviewing and history taking, problem solving and patient management) were assessed in this study. The percentage range of scores was 0–100. We double rated all available consultations and followed the methods described in the LAP and previous work on assessing recorded consultations.[16,17] The recordings were independently reviewed with reference to the LAP categories by two investigators (GM and MJ). The scores were then compared and the final scores represented the consensus view on the quality of the consultations. The value of the LAP is that it measures the quality of the consultation independently of the problems presented.
Stimulated recall session
The practitioners and actor-patients were shown brief episodes of selected consultations and asked to comment on:
1) how the NAT-C impacted on the consultation and
2) how it could be deployed to best effect in the context of a subsequent formal clinical evaluation.
Statistical analysis
As the data were not normally distributed the Kruskal–Wallis test was used to examine the differences in the LAP scores between GPs and between scenarios. We applied a GEE model to take into account the dependence of LAP scores (GP) and the different scenarios. The correlation between the LAP score and minutes in consultationwas measured using the Spearman’s correlation coefficient.
Thirty-four consultations were successfully recorded. The team reviewed the performance of the actorpatients and confirmed that there were no remarkable differences in the way that cases were portrayed to the practitioners. Two doctors opted for audio rather than video recordings. The mean duration of the consults was 13 min. 47 sec. (range 6 min. 3 sec. to 22 min. 51 sec.). One doctor’s audio recordings failed in two cases. This provided a total of 19 consultations in which the NAT-C was and 14 in which the NAT-C was not used for review.
GP core competencies
There was evidence of significant differences between GPs in core competencies as measured by the LAP (P<0.001). One doctor, shown as Doctor 2 in Table 1, demonstrated exceptional skill, and Doctor 6 showed much less skill than the others.
There was no evidence of differences in doctors’ performance according to scenario (P = 0.99). The scores according to scenario are presented in Table 2.
Impact of the NAT-C
LAP scores were higher when the NAT-C was used compared to when it was not used, as shown in Figure 1. The use of NAT-C improved LAP score by an average of 3.3 (95% CI: –3.99, 10.6) points on the LAP score, after controlling for the different GPs and scenarios. However, this improvement is not statistically significant (P=0.37).
Figure 1: Impact of the NAT-C on the LAP score per scenario
We were able to plot the LAP scores in each consultation and relate them to the number of minutes into the consultation before the doctor reviewed theNAT-C. Figure 2 suggests that there may be a slight decreasing trend in LAP score as minutes increase (r = –0.25). However, there was a heavy influence of two extreme values (9 and 13.5 minutes). The number of minutes that elapsed before the NAT-C was introduced was not a predictor of LAP scores after GP and scenario were adjusted for (P = 0.92). The LAP score increases by 0.03 (95% CI: –0.64, 0.71) points for every one minute increase.
Figure 2: Relationship of LAP score to minutes in the consultation before GP reviewed the NAT-C. Demonstrating that there may be a slight decreasing trend in LAP score as minutes increase (r = –0.25)
Refining the NAT-C
Doctors and actors made several recommendations about how the NAT-C could be improved. Box 2 consists of a summary of their recommendations. The general consensus was that the NAT-C was a helpful intervention. It allowed the ‘caregiver patients’ to voice their concerns, focused the consultations on specific areas that might not have been addressed without a prompt and in the case of the patients allowed them to feel that they had ‘permission’ to raise issues which were not necessarily directly related to an ongoing physical medical problem. However, the participants also made recommendations about how the NAT-C could be further refined for use in practice. The details of the NAT-C will be reported separately.
This study aimed to pilot test an intervention prior to conducting a formal randomised controlled trial. Whilst consultation performance was somewhat improved in consultations which referred to the NAT-C, these improvements were not significant. Importantly, the pilot study identified specific areas to further refine the NAT-C. The approach taken to developing this complex intervention had several strengths. We were able to replicate conditions that may be difficult to control in clinical practice. The practitioners all consulted the same patients in the same sequence. The doctors demonstrated a range of consultations skills, as was as expected since the doctors came from different backgrounds and had different levels of experience. There was no evidence of differences in doctors’ performance according to scenario. This suggests that all the scenarios were equally testing of practitioner skill. The use of actors as standardised patients to assess the performance of healthcare professionals during faceto- face consultations has previously been reported. This method has been shown to be a reliable and valid way to assess clinical performance.[21] In many ways the methodology involving consulting actor-patientsmimics the formal assessment or examination of candidates seeking membership to many professional colleges.
Participating GPs were volunteers and as such may not have necessarily been typical of their colleagues. It is possible that those who volunteered may have had better than average consultation skills or an interest in the topic of the research. Recruitment is a particular challenge in labour intensive studies especially when they require observation of consultation practice, however, where the participants are also providing guidance on the development of an innovation we believe the advantages of involving interested and willing volunteers outweigh the need to recruit a ‘representative’ sample. Nonetheless we have no measures of how the volunteer practitioners perform in routine practice without the study using the LAP or any other measure of competence. We are, therefore, unable to confirm how well their performance here reflects their competencies when consulting ‘real’ patients. For the purpose of this study we had hypothesised that the NAT-C could have disrupted the flow of the consultations – the ‘switch costs’ of the intervention as predicted by Rubinstein et al.[12]While we had no direct measure of this, had it been a problem, we would have expected it to be reflected in the LAP scores whichwere allocated.While this was encouraging and helpful data, we cannot exclude the possibility of a clinically important negative impact. Therefore we emphasise that this method is a prelude to, but not a substitute for, a formal randomised trial in clinical practice.
The clinical challenge and the innovation
We hypothesised that the administrative tool may have heightened the practitioners’ inattentiveness to other important clinical issues which were included in the scenarios but not mentioned on the NAT-C, thus reducing the LAP scores. Neither practitioner examination skills nor the impact of that examination could be assessed in this experiment. Time taken to undertake a physical examination might also have increased the duration of the consultations substantially. However, it may be impractical and is possibly unethical to subject actors to intimate physical examination. In this study it might have been helpful for the actorpatients to present ‘incognito’. However, as has been noted previously, fielding incognito standardised patient visits as known patients is feasible but labour intensive, and within the constraints of the project reported here which included a requirement to record consultations, incognito presentations were not practical. [22]
The scoring of consultation competencies
In a previous study using actor-patients itwas concluded that a professional media team should be employed, guaranteeing high quality footage, with the least disruption or inconvenience to the participants.[16] We were able to confirm this impression in this study as the majority of consultations were captured with sufficient technical clarity to be available for analysis. However, we were unable to assess the impact of observation on the GPs’ performance, although the literature on video recording for the purposes of assessment suggests that it has no significant adverse effect. Agreement by assessors on GP LAP scores was generally good. The assessors were from similar practice backgrounds (UK and Australia) and seniority (25 and 15 years), but differed in experience of assessment (one study vs. three studies). Close attention was therefore paid to achieving a consensus score. Finally, as investigators in the study the assessors could not be blind to the aims of the study. We do not believe this had an impact on the scores. However, it would be prudent to deploy assessors who could be successfully ‘blinded’ at the time of reviewing the consultations.
We present work on developing an intervention prior to conducting a formal randomised controlled trial. The intervention as conceived appears not to harm core competencies. The pilot study also suggests that there was scope to further refine the NAT-C. Had the intervention been tested in a formal trial without piloting in the context of the consultation there might have been significant difficulties with administering the intervention in practice.
This study received ethics approval from HREC at Curtin University of Technology (RD-35–08).
Not commissioned; externally peer reviewed.
None.
