Research Paper - (2009) Volume 17, Issue 3
Dionne Sofia Kringos MSc*
International Health Services Researcher
Wienke Boerma PhD
Senior Researcher International Health Services
NIVEL – Netherlands Institute for Health Services Research, Utrecht, The Netherlands
Martina Pellny MSc
Programme Officer for Primary Health Care, WHO Regional Offce for Europe, Copenhagen, Denmark
Received date: 27 January 2009; Accepted date: 29 March 2009
Background This World Health Organization (WHO) study aimed to develop and field test an instrument to assess the availability of structures and mechanisms for managing quality in primary care in countries in transition. Method The instrument is based on a literature study, consensus meetings with experts, and observations in these countries. It consists of three parts: a semi-structured questionnaire on national policies and mechanisms; a structured questionnaire for general practitioners (GPs); and a structured questionnaire for use with managers of primary care facilities. The instrument has been field tested in 2007 in Slovenia and Uzbekistan. Results In Slovenia, leadership on quality improvement was weak and local managers reported few incentives and resources to control quality. There was a lack of external support for qualityimprovement activities. Availability and use of clinical guidelines for GPs were not optimal. GPs found teamwork and communication with patients inadequate. In Uzbekistan, primary care quality and standards in health centres were extensively regulated and laid down in numerous manuals, instructions and other documents. Managers, however, indicated the need for more financial and nonfinancial levers for quality improvement and they wanted to know more about modern healthcare management. GPs reported strong involvement in activities such as peer review and clinical audit, and reported frequent use of clinical guidelines. Overall, the information gathered with the provisional instrument has resulted in policy recommendations. At the same time, the pilot resulted in improvements to the instrument. Conclusion Application of the instrument helps decision makers to identify improvement areas in the infrastructure for managing the quality of primary care.
healthcare evaluation mechanisms, healthcare quality assurance, primary health care, Slovenia, Uzbekistan
Healthcare systems should have inbuilt mechanisms that allow a monitoring of the quality of services provided.[1,2] Policy objectives to improve the quality of care result from a more general requirement that health systems are cost-effective.[3,4] Strong primary care (PC) is supposed to enhance the cost-effectiveness of the system as a whole. Strong PC refers to easy access to first-contact services, a comprehensive supply of curative, preventive and rehabilitative services, conti-nuity of care, and co-ordination with other PC pro-viders and other levels of care.[5–9]
Many studies have pointed to large variations in the quality of PC services and in providers keeping to accepted standards.[10,11] Since PC is usually delivered in small and relatively independent units, quality assur-ance is more diffcult to organise.[12,13] In developing healthcare systems, ‘quality awareness’ is usually low and mechanisms for maintaining and improving health-care services are not well developed. The development of a strategy for quality improvement and the imple-mentation of mechanisms to routinely provide feedback information on the quality of facilities and health services is often part of health sector reforms in these countries.[14]
The WHO Regional Office for Europe
The World Health Organization (WHO) Regional Of-fice for Europe supports member states to strengthen their health systems.[15] The World Health Report 2008 urged countries to act on the evidence that access to PC services should form the core of appropriate health-care systems.[16] Individual member states are supported to develop, among other things, strategies and mechan-isms for systematic quality improvements by means of Biennial Collaborative Agreements (BCAs). WHO’s initiative to support development of the instrument presented in this paper fits well into this policy. Box 1 provides background information on the PC mission of WHO Europe.
Objective
This article aims to describe how an instrument to assess strategies and mechanisms for quality assurance of PC staff and services has been developed and to present the results of a test of the tool, in Slovenia and Uzbekistan.
Study design
A full description of the study design and development process of the instrument has been reported else-where.[17,18] This section therefore provides a sum-mary.
The study took place in 2007/2008 and started with a literature review to identify key functions and exist-ing instruments to measure quality management in PC. This resulted in a typology and checklist for quality-improvement policies and activities. The results were discussed in a meeting with 14 policy makers from ministries of health from five countries, re-searchers from NIVEL, and representatives from WHO. Participants validated the initial ideas and provided the researchers with country-specific information. The next step was the development of the draft instrument, consisting of three questionnaires (one for the national level; one for managers in PC and another for general practitioners (GPs)). The draft was revised after the researchers had visited the countries selected for the pilot implementation. The questionnaires were trans-lated into the Slovenian and Uzbek languages. The fieldwork, jointly conducted by a local co-ordinator and researchers, included the sampling procedure, training of field workers, distribution and collection of questionnaires and organisation of data entry. Analysis and reporting were carried out by the re-search team in the Netherlands, presenting results,experiences with the instrument and recommenda-tions for its future use.[17,18] At an international review meeting, organised by WHO, 34 primary care experts (including researchers, policy makers from ministries of health, academics and consultants) from 14 countries discussed the provisional results and evaluated the instrument.
Countries and regions for the pilot
The pilot study took place in Slovenia and Uzbekistan. In Slovenia, the capital Ljubljana and the relatively rural region of Gorenjska were selected by way of contrast. In Uzbekistan, the provinces of Fergana, Syrdarya and Tashkent (excluding the capital) were appointed as pilot areas because these were in different stages of PC reform.
Sampling and data-collection strategy
Details of the study population and data collection are summarised in Table 1. In Slovenia the directors of PC units and the heads of family physicians in the PC facilities were included as managers. GPs were recruited from both public and private practice. In Ljubljana city as well as in Gorenjska the total population of managers and GPs were included. Following advice by local investigators, questionnaires were distributed by post, and followed up by telephone reminders.
In Uzbekistan, the target population of managers were the deputy head district doctors. All managers were included. The population of primary care phys-icians included GPs who had completed a retraining programme as well as doctors who had not. In the provinces mentioned, random samples were drawn from alphabetical lists of GPs. In order to have approx-imately equal numbers in each region, a 20% sample was drawn in Fergana; in Syrdarya and Tashkent 50% samples were taken. GPs received a questionnaire and a sealable envelope via their manager. The freedom to participate and confidentiality were stressed.
To answer the national-level questionnaire, in both countries panels of experts were formed, consisting of representatives from the Ministry of Health and stake-holder groups (such as medical associations, health insurers and academics).
Data processing and analysis
For data entry, SPSS Data Entry Station version 3.0.3 was used. The program was installed and explained to local staff responsible for data entry. For analysis, SPSS version 14 was applied.
The Primary Care Quality-management Instrument (PCQmI)
For quality management to be embedded in healthcare systems, various functions, related to different parties, need to be activated, such as stewardship or govern-ance; advocacy; facilitation and advising; implementa-tion; teaching and training; monitoring and evaluation; research; communication among stakeholders.[2,4] The PCQmI has focused on the state of institutionalisation of these functions in PC systems. It aimed to identify the currently available structures in a country and possible areas of improvement and, thus, enable deci-sion makers to set priorities for further development of quality systems in PC. The instrument focused on different levels in the healthcare system: regulation and structures at the national level, the management of PC facilities and the providers of care – the physi-cians in PC. Table 2 shows for each questionnaire the topics that were addressed.
Experiences with the instrument
The questionnaires have been revised as a result of experiences and feedback during the field tests and comments made by the experts in the Copenhagen meeting organised by WHO (the latest version of the structure of the three questionnaires can be requested at www.nivel.eu/who). In general, questions have become more factual. The character of the national-level questionnaire was drastically changed into a tem-plate for a background document. These backgrounds were to be prepared by a small panel of experts, and subsequently discussed in a national validation meet-ing. Questionnaires for managers and GPs have been reduced in size. Furthermore, it was advised to im-prove the sensitivity of the instrument by supple-menting it with additional document inspection and site visits.
For reasons of comparability between countries and within one country over time, the importance of uni-formity was stressed. However, to allow for local priorities, an optional variable annex to the generic core of the instrument would strengthen its applicability.
Slovenia
Response
Out of 17 invited experts, ten effectively filled in the national level questionnaire, and only five of these participated in the consensus meeting. Among the included and approached managers about half com-pleted the questionnaires (nine in Ljubljana and five in Gorenjska). On average, managers in Ljubljana had been working in this position for 27 years, in Gorenjska for 13 years.
The response among GPs was low. Only 63 GPs in Ljubljana returned the questionnaire (48%) and 18 in Gorenjska (26%). Three-quarters of the GPs were female. The average age in both areas was around 47 years. About three-quarters of the GPs, in both Ljubljana and Gorenjska, had completed a postgradu-ate training in family medicine. On average, the GPs had well over 20 years of working experience, most of the time at the place they were currently working.
Quality assurance in Slovenia: national level
Table 3 provides an overview of main results on the national level, based on opinions of PC experts.
Quality assurance was not a priority in PC in Slovenia. At different levels, leadership and a clear vision on maintaining and improving quality of ser-vices were lacking. Legislation was proceeding slowly. At the time of the study, three laws relating to quality in health care were pending. The system was not pre-pared for accountability, illustrated by the rare use of external quality assessments commissioned by the Ministry of Health, low use of available public health data, and a lack of supervision on complaint pro-cedures. Beyond the formally arranged inspection and supervision in health care, supervision of quality in PC was fragmented and poorly co-ordinated. The con-tinuing medical education (CME) system was based on credit points, and not driven by the educational needs of physicians. Access to guidelines and protocols could have been improved. Independent guidelines were for sale only. Those provided by the pharma-ceutical industry were freely available, but less suitable for use in PC.
PC managers in Slovenia
Table 4 provides a selection of proxy indicators for managers in both pilot areas in Slovenia.
Annual quality reports were unusual. There was a low use of formal quality-assessment instruments, particularly in Ljubljana, such as attestation of phys-icians, voluntary certification and accreditation, and mandatory licensing of physicians or nurses and organ-isations. Internal assessment mechanisms, such as routine inspection of medical files, were not generally applied in all centres. In contrast to managers in Gorenjska, those in Ljubljana generally rated the con-ditions and means available for quality improvement as insuffcient. In both regions, human resources man-agement was insuffciently suited to quality improve-ment. Only a minority of the managers (20% in Ljubljana and 40% in Gorenjska) reported offering staff training for quality improvement, for using personal develop-ment plans and to monitor job satisfaction of staff. Managers agreed with the statement that a more positive attitude of staff towards innovation was needed. Proto-cols and guidelines were not generally implemented – only in about half of the centres. Managers expressed their intention to invest in further implementation, and to update obsolete procedures. However, they reported they were confident that patients were treated according to the latest professional evidence.
Slovenian GPs
A selection of indicators concerning GPs in Slovenia is shown in Table 5. GPs were more involved in ad hoc forms of quality improvement, than in structured and formalised procedures. Clinical guidelines were not optimally used. A co-ordinated approach was missing in the production of clinical guidelines. GPs were positive about CME courses, stating that these helped them to provide better care to their patients. About 75% of the GPs in both regions saw opportunities to improve teamwork and co-operation within PC, for example with nurses, as well as in the interface with secondary and tertiary care. GPs widely acknowledged that the motivation of healthcare workers to improve the quality of care left something to be desired and that better incentives would help to change the situation.
Recommended improvements in Slovenia
Although the main aim of the pilot was to test the implementation of the instrument, the results give rise to suggestions for decision makers. The recommen-dations listed in Box 2 have been formulated by the authors on the basis of the results of this pilot application and their experience with primary care development in countries in transition.
Uzbekistan
Response
All 11 invited experts effectively participated in the national consensus meeting. All 40 PC managers in the three provinces responded by filling the questionnaire(16 in Fergana, nine in Syrdarya, 15 in Tashkent). Most of them were male and had been working in this position and this centre between 10 and 20 years. In the densely populated Fergana province, more than two-thirds of the managers worked in inner-city or suburban areas, while in Syrdarya three-quarters were working in suburbs or small towns and in Tashkent region 60% in rural areas. The response among GPs was close to 100% and amounted 106 GPs in Fergana, 97 in Syrdarya, and 103 in Tashkent. Overall, 42% of the GPs were male and 58% female. The average age of the GPs was 44 years. Most GPs had completed a retraining course in family medicine. Since GPs were relatively new in Uzbek PC, respondents had little experience as a GP, but much more as a paediatrician or therapist. As the introduction of GPs in PC started in the countryside, the large majority of GP respon-dents were working in rural practice.
Quality assurance in Uzbekistan: national level
Table 3 provides an overview of main results on the national level based on opinions of PC experts. With donor support, quality improvement in PC was an explicit national priority. Many laws, decrees and orders dealt with the improvement of (primary) healthcare services. Reforms also aimed to improve healthcare management. The Evidence Based Medicine Centre was in charge of the development and implementation of a programme for clinical guidelines in PC. The final responsibility for the quality of PC was with the Ministry of Health, but within this ministry responsi-bilities seemed to be fragmented. The position of non-governmental organisations (NGOs) in healthcare mat-ters seemed to be weak. Promoting patient-centred care was not a policy priority. Major topics in the Law on Patients’ Rights were compliant procedures, patients’ informed consent and patients’ access to their medical files.
PC managers in Uzbekistan (see Table 4)
The availability of quality-related documents (e.g. mission statements, or budget specification) was clearly better in Fergana than in Syrdarya or Tashkent. Internal assessment was fairly practised. In all three provinces, 50 to 75% answered they used evaluation reports, internal medical audits, GP peer review, and quality- improvement committees. Patients’ needs were infre-quently monitored.
Availability of internal resources for quality assur-ance, such as management information, support and incentives, seemed to be best in Fergana, followed by Syrdarya. In Tashkent, most managers found these resources to be insuffcient. Furthermore, the man-agers indicated they needed more support for quality improvement and more modern management infor-mation and skills. Managers found the attitude of healthcare staff towards innovation to be an obstacle for quality improvement.
Uzbek GPs (see Table 5)
GPs were confident that treatment of patients was in line with the latest evidence, while managers were more reserved at this point. GPs, especially in Syrdarya and Fergana, reported they were highly involved in formal and informal quality-improvement activities. In Fergana and Syrdarya, clinical guidelines were generally used, while in Tashkent some improvement seemed possible. Like the managers, the GPs found CME courses to positively contribute to the quality of care. GPs were more strongly convinced than their managers that they spent suffcient time to keep up to date. GPs in general expressed the intention to im-prove many aspects of their clinical work, such as diagnostics, and drug prescriptions, but they found the style of management to be punitive rather than stimulating.
Recommended improvements in Uzbekistan
Recommendations made by the authors to improve PC quality management in Uzbekistan are listed in Box 3.
Evaluation of the implementations
The involvement of committed counterparts and local experts turned out to be crucial for a successful imple-mentation of the instrument. The surveys had a wider impact than just collecting data. The introduction of the activities at central, regional and local level implied information transfer and raising awareness on issues of quality in PC. The more intensive the approach and the more personal the surveys that were introduced, the stronger this effect has been. Several data-collec-tion methods can be identified for the surveys: postal or personal; via the lines of management or parallel. Whichever method is chosen depends on available resources and local circumstances.
A deliberate choice of pilot areas is important. Preferably, these should be contrasting in variables the instrument aims to measure, for instance in the stage of healthcare reform or because of different provision of PC services. The formulation of differences between regions may serve as a reference for the interpretation of results and offer a starting point for follow-up activities.
Limitations of the pilot
The instrument relies on self-reports, rather than on direct observations or registrations. The draft instru-ment was revised to reduce the likelihood of bias as a result of a positive answering tendency. The accept-ability of the instrument influences the response rate. This was relatively low in Slovenia, which could be for two reasons. Firstly, physicians were approached by postal questionnaires, whereas in Uzbekistan a more personal approach was used. Secondly, independently practising physicians have more freedom to reject an intervention, compared to physicians practising in a command-control system, such as in Uzbekistan. To compensate for possible low response rates, which was the case in Slovenia, additional observations and inter-views have been included in the revised instrument. Furthermore, it should be stressed that the instrument is not about quality of care itself or quality indicators. Since follow-up of the formulated recommendations is still to come in both countries, an evaluation of this process is missing in this paper.
Application of the instrument
The instrument aimed to get insight into available strategies and mechanisms on quality assurance and the way managers and practitioners are dealing with quality assurance. Since this varied information is not readily available, especially not in countries in transition, the questionnaire method was considered to be the most appropriate approach. The use of surveys implemented by national counterparts, discussed and completed by diverse stakeholders, furthermore had the advantage of raising awareness on the importance of quality management. In a relatively easy way, the surveys also produce information that allow decision makers to identify areas of improvement. The involve-ment of stakeholders may strengthen their commitment related to quality management in PC. The catalysing role of WHO in this process is essential to move this process forward. Together with national authorities and stakeholders, workshops and conferences are organ-ised to disseminate results and transfer expertise for follow-up.
The pilots will result in a new revised version of the instrument, which in the future can be implemented in other countries. Implementation in new countries can take place in the context of a BCA between the respective ministries of health and WHO.
Application of this new instrument in the context of WHO country activities can enable decision makers to identify areas of development in assuring the quality of PC services. Applicability can be enhanced by tuning the generic instrument to the local situation before use.
The research team gratefully acknowledges the finan-cial support of the WHO Regional Offce for Europe.
None.
Not commissioned; externally peer reviewed.
None.
The authors acknowledge the WHO Regional Offce for Europe for initiating and funding the study. They also express their gratitude to the managers, phys-icians, local organisers and others involved in primary care in the provinces of Fergana, Syrdarya and Tashkent (Uzbekistan), and in Ljubljana and the Gorenjska region (Slovenia). Valuable inputs by the participants at the review meeting organised by WHO Europe are highly appreciated.
