Research Paper - (2013) Volume 21, Issue 1
Department of Family Medicine, Mayo Clinic, Rochester, MN, USA
Received date: 16 October 2012; Accepted date: 18 December 2012
BackgroundCase–control designs are rarely employed in quality improvement investigations. The reason for the study was two-fold: (1) to demonstrate the feasibility of the method in a practice setting, and (2) because drop-outs from this programme had not previously been investigated. MethodsDrop-out was defined as termination due to non-contact after two consecutive months. Fifty drop-outs were randomly selected from all cases discharged from the programme during the period 1 March 2008 to 28 February 2012. Fifty controls were randomly selected from among those discharged patients who did not drop out due to non-contact. ResultsDropping out was significantly associated with female sex (P = 0.015), younger age (P = 0.000) and treatment site (P = 0.004). Sites still in the shake-down period had higher drop-out rates. Depression diagnosis and severity were not significant. ConclusionsThe case–control design is an efficient approach to retrospective analysis of discrete quality indicators.
case–control design, collaborative care management, depression treatment, integrated behavioural health care, managerial epidemiology, primary care
How this paper fits in with quality in primary care
What do we know?
Collecting data for quality assessment can be difficult even with electronic medical records. This report addresses methods for efficiently and validly conducting quality assessment studies.
What does this paper add?
We show that a case–control design can be efficient because it allows for smaller samples and the data can be readily analysed.
Previous studies of the outcomes of collaborative care management (CCM) for the treatment of depression in primary care patients have employed standard epidemiological approaches typically found in retrospective studies of medical records.[1–3] A large sample is drawn, thus enabling investigators to develop adjusted odds ratios or linear regression coefficients. Extracting large data files has several disadvantages for the investigator. Records may not be digitally recorded, precluding automated queries. When records are searchable, programming expertise may not be available. New programmes may not have treated enough cases to allow for epidemiological modelling. Also investigators may lack access to biostatistical support for the analysis of large data sets.
Case–control designs might solve some of these problems. Applied most often to the analysis of disease cases compared with subjects not having the disease, the case–control design can be converted to the analysis of adverse events, thus becoming a quality improvement methodology. Cases and controls can be easily extracted manually from medical records and entered into a spreadsheet, lowering the barriers facing the investigator.
However, published quality improvement studies employing case–control designs are difficult to locate. Use of epidemiological methods to management problems such as quality control can be described as managerial epidemiology. Managerial epidemiology is a required subject in healthcare management programmes [4] and several textbooks have been written in this field.[5–7] Nevertheless, a keyword search of the medical literature finds no managerial epidemiology studies. Clearly, the field of managerial epidemiology is underdeveloped.
The purpose of this study was to test the usefulness of an epidemiological method (the case–control design) for addressing a management concern (quality improvement). The quality indicator employed was the drop-out rate. Previous studies of the outcomes of CCM have analysed clinical outcomes such as remission and persistent depressive symptoms after six months.1–3 Drop-out rates have previously been studied as outcomes of behavioural health programmes. [8,9] However, this report is the first to study drop-out rates of CCM programmes.
The reason for the study was two-fold: (1) to demonstrate the feasibility of the method in a primary care practice setting, and (2) because drop-outs from the CCM programme had not previously been investigated. CCM is an approach to organising clinical services that relies heavily on care managers who monitor patient progress via telephone. Psychiatrists and psychologists assess and diagnose but their primary function is consultation to primary care providers and care managers.
In our practice, CCM originally started in 2008 at one clinical site and, by 2010, was implemented at all five of our clinical sites. By February 2012, the programme had been in operation for three years in two locations and for one year in three other clinic locations. One thousand, three hundred and twenty-six patients were enrolled during this time frame.
A description of the CCM process and its implementation is reported in our prior studies.[3,10] Enrolment into our programme required a clinical diagnosis of major depression or dysthymia and a Patient Health Questionnaire (PHQ-9)11 score of 10. The minimal exclusion criterion was the psychiatric diagnosis of bipolar disorder.
For the purposes of this study, drop-out was defined as termination due to non-contact after two consecutive months. Fifty drop-outs were randomly selected from all cases discharged from the programme during the period 1 March 2008 to 28 February 2012. Fifty controls were randomly selected from among those discharged patients who did not drop out due to non-contact. The random selection procedure was as follows. All of the available cases were assigned a random number in the Excel spreadsheet. The random number was copied and pasted as ‘values only’ so as to fix the random number. The data set was sorted by random number. The first 50 dropouts and the first 50 controls were selected for the study. This study was approved as a modification of a previous protocol by our local institutional review board.
Cases and controls were compared in regard to treatment site, sex, mean age and diagnosis. Three diagnostic categories were included: first (ICD9 296.2) or recurrent (ICD9 296.3) episode of major depression, or dysthymia (ICD9 300.4). Intake PHQ-9 score was used to assess depression severity. The chisquare test was used for sex, diagnosis, and treatment site and Student’s t-test was used for age and PHQ-9. Fisher’s exact test was used for the dysthymia diagnosis.
Results of significance testing are shown in Table 1. Drop-out was significantly associated with female sex (P =0.015), younger age (P= 0.000) and treatment site (P = 0.004). Baseline PHQ-9 and diagnosis were not significantly associated with caseness using P < 0.05 as the criterion. The first sites implementing the CCM programme had the most cases. The three newer sites, still in the shake-down period, had higher drop-out rates. Drop-out rates were significantly lower in more established locations (P = 0.004).
Table 1: Comparison of drop-outs and controls (n = 100)
Quality control is a management problem. Application of epidemiological methods to management concerns can be defined as managerial epidemiology.7 Therefore, this study, which applied the case–control design to a quality control problem, is a relatively rare example of managerial epidemiology.
This approach is well suited for use in primary care organisations. With the advent of the primary care medical home and its attendant emphasis on evaluating the quality of care processes,12 this model of quality assessment could be used to evaluate a wide variety of healthcare delivery initiatives in order to identify potential areas to concentrate on for improvement. Examples of where this method could be applied are myriad and include evaluation of collaborative care models for chronic disease management, preventative care initiatives and care coordination. There are several potential advantages of this approach. Investigators who have limited access to automated queries of electronic medical records would find that coupling this design with a small sample reduces the cost of data extraction and accelerates data collection time.
When adverse events are rare, the case–control design balances the number of cases and controls and thus increases the power to detect significant associations. Significant variables can become the focus of interventions designed to reduce the rate of adverse events. System corrections need not wait for collection and analysis of large samples.
Finally, we note that studying small samples with a case–control design simplifies data analysis. Statistical tests can be performed in a spreadsheet and analysis does not require technical support from a biostatistician.
Disadvantages of this application of the case–control approach arise from the small sample.Wechose to only extract 100 cases for this test of the method (50 cases and 50 controls). This small sample precluded preparation of a run chart depicting variations in drop-out rates by month. Investigators seeking to apply the case–control approach in a real-time situation would find themselves increasing the sample size in order to achieve sufficient cases in each period. The larger sample would make the case–control design unnecessary. The approach is most useful for retrospective studies.
The small size of our sample also limited the potential for multivariate modelling. Development of adjusted odds ratios would have required more than the 100 cases in our sample. We note, however, that univariate tests supplied useful information about patient-specific and site-specific predictors of higher drop-out rates.
The case–control design is observational and as such may be subject to biases not found in randomised experiments. A cohort design would also be preferable. Case–control designs often are employed due to their increased efficiency, particularly when the outcomes studied are rare events. Confirmatory studies usually find results from case–control studies to be accurate, but our findings should be replicated in other sites before they are widely accepted. Another weakness of the design is not matching on age and sex. Matching is typical in case–control studies, particularly when the epidemiologist is able to electronically extract cases from a large data base. Quality improvement analysts may be compelled to manually extract cases from a smaller pool of records, so we sought to test the method with a small number of cases: we arbitrarily chose 50 drop-outs and were pleased to obtain useful results. Even so, matching on age and sex would have strengthened the conclusions.
A third limitation of the study is the restricted amount of clinical information. We focused on the PHQ-9 because this is the measure primary care providers rely upon in our practice. Because this is a management study conducted in a field setting rather than a funded psychiatric research project, we feel this decision is appropriate.
Fourth, unmeasured differences between the sites could easily have affected the results.We observed that drop-out rates were lower in sites that implemented the programme earlier.However, we cannot be certain that other factors might have been at work, including negative attitudes toward psychiatric services among patients or providers.
Finally, the small number of cases precludes multivariate analysis. This limits the number of variables that can be controlled. However, it also demonstrates that a case–control design can achieve useful findings with a small sample.Wenote that the purpose of this study was to demonstrate the feasibility of applying the case–control design to a practical quality improvement question in a primary care practice setting, which we feel has been accomplished.
This study tested several potential predictors of drop-out rates from theCCMprogramme. Significant patient factors included sex and age. Clinical variables, diagnosis and baseline PHQ-9 were not significant in this sample. Significant site-specific differences were revealed. The programme was launched initially in two sites and later rolled out to three additional sites. The newer sites were revealed in this analysis to have higher drop-out rates. This is not surprising because they were still in the shake-down phase of starting a programme that was new to them. We can report parenthetically that by the end of 2012, drop-out rates were declining in the newer sites.
The case–control design, retrospectively applied as we did in this study with a small number of cases, allowed for efficient measurement of variation in drop-out rates and identification of significant patient- and sitespecific predictors. The results are encouraging andwe conclude that the approach should be tested with other programmes, with other quality indicators, and in other clinical practice locations. Our findings also suggest that increased use of epidemiological methods to address managerial problems in healthcare organisations might be practical and useful.Quality managers in primary care should consider testing the feasibility of the case–control design in their practices.
Mr Isaac Johnson and Ms Julie Maxson assisted with abstraction and collection of the data.
Department funds were used for this manuscript.
The study was approved by the local institutional review board.
Not commissioned; externally peer reviewed.
None.
