International Exchange - (2004) Volume 12, Issue 1
Researcher, Quality Division, Center for Health Service Management, and Lecturer, Department of Public Health, Faculty of Medicine
Adi Utarini MSc MPH PhD
Chief Consultant, Quality Division, Center for Health Service Management, and Senior Lecturer, Department of Public Health, Faculty of Medicine
Tjahyono Kuntjoro MPH DrPH
Senior Consultant, Quality Division, Center for Health Service Management, and Head, Gombong Health Training Centre
Gadjah Mada University, Indonesia
Accepted date: 25 June 2003
This paper is a report of a process we undertook to develop standards for accreditation of primary care mservices in Indonesia. The process consisted of an appraisal of published standards and convening meetings of stakeholders to discuss and agree the standards for use in Indonesia. A literature review informed the process. The evaluation exercise consisted of assessment of the ‘structure, process and outcome’. Lack of patient involvement was identified as a key omission. Other critical areas that need to be addressed before the standards can be recommended for wide scale dissemination and implementation were also identified.
accreditation, evaluation, Indonesia, standard
There has been growing recognition worldwide that measurement of quality of care is a critical building block for system-wide improvement of healthcare and patient outcomes.[1,2] However, the evaluation of qual-ity is a complex task and requires specification of standards and their measurement.[3] There is a need for explicit statements which define quality to enable precise measurements.[4] These statements in the liter-ature are commonly referred to as guideline, protocol, or, as the term used in this paper, ‘standard’.
Whether the use of standards will ultimately lead to quality improvement remains a concern.[5] If quality is assessed against inappropriate criteria then there is a risk that resources may be wasted and false improve-ment may be seen. This causal relationship between standards and improvement has been observed to be dependent on many factors, including the quality of the standards, and the extent to which these are meaningful, valid and interpretable.[6]
The formal evaluation of standard validity – the ability of the standard to bring about the anticipated outcomes when adhered to – would be useful for this purpose.[7] Such evaluation, however, is highly demanding and resource intensive. This type of evalu-ation is not widespread.[8] An assessment of the rigour within which standards are created would be the most practical way for evaluation in the absence of outcome indicators to judge e¡ectiveness.[7] The strategy in essence is to critically appraise the standards, deter-mining whether developers have been rigorous in minimising potential biases. This method is similar to critical appraisal of research. However, the devel-opment of this systematic appraisal approach to standards is in its infancy.
The Ministry of Health, Republic of Indonesia recently established the Directorate for Primary Care with the mandate of standard development and regulation of public and private primary care services.
The Centre for Health Service Management, at Gadjah Mada University (CHSM-GMU) has been contracted to provide the directorate with technical support for designing accreditation standards. This paper will report our experience of the primary care services accreditation standards project and demonstrate the potential of the method which we are calling the ‘appraisal method’ in a resource-constrained setting.
The appraisal protocol was developed from a liter-ature review. References were drawn from literature published between 1992 and 2002, identified through electronic search using Medline and Business Search Elite, based on the following keywords individually or in combination: healthcare, quality, standards, per-formance, measure, guideline, and indicator. Addi-tional references were sought by: browsing through the library collections of Gadjah Mada University (Indonesia) and Prince Leopold Institute of Tropical Medicine (Belgium), consultations with experts in quality, and retrieving references cited in key publica-tions. Data related to structure, process and output attributes of standard, guideline or indicators were extracted. Expert review was undertaken to ensure face validity of this protocol. The data sources for the evaluation consist of direct involvement in the project and archives (audio recordings, minutes, o¤cial documents).
The structure for standard development
There appears to be reasonable consensus in the references that the group involved in the development process should be multidisciplinary and include representatives from relevant stakeholders.[9] In this case, a number of stakeholders were integrated in di¡erent stages of the process. Mainly sta¡ from the Directorate (proposed user – regulator) and CHSM-GMU (academics) were involved initially. The aca-demics consisted of experts in quality management, clinical epidemiology and health economics. Subse-quent development phases incorporated other stake-holders, i.e. provincial health o¤ces, district health o¤ces, primary health centres and the National Hospital Accreditation Commission. The involve-ment of regulators and those who would be assessed (providers – primary health centres) assured ownership of the standards and use in practice.[6,8] However, throughout the development stages, there was no representation of patients or patient advocates. Advice from this group of stakeholders is crucial.[10,11]
We assembled a project team consisting of a leader, panel of experts, research assistants and administra-tive support sta¡. There were no explicitly defined roles and responsibility. This indication of a relatively
• uid system is in line with the general observation that the development process is rarely systematic and structured, regardless of the method applied.[12] Such
• exibility increases the likelihood that factors other than scientific evidence and balanced contextual information will be brought to bear in the develop-ment process. However, formal methods such as interaction process analysis would be needed to allow an objective judgement.
The project document consisted of terms of refer-ence describing background, aim, specific objectives, methods and participants. Workshops and team meetings were planned to serve as an interdisciplinary consensus development forum. Key project planning elements had been elaborated in su¤cient detail, yet some of the specific elements suggested by the Inter-national Society for Quality in Health Care to be covered in such a document were missing, e.g. market requirements and opportunities, institutional stra-tegic directions, resources, statutory requirements, views of interests.[11] Incorporation of such elements would arguably have improved the process.
The methods used in the standards development process
The key concept of assessing quality lies in evaluating only those processes of greatest significance to the situation under review.[13] A good standard develop-ment process should be able to demonstrate adequate e¡orts in choosing the most important areas by having clear prioritisation criteria. The exercise in this case started by mapping established primary care accred-itation standards, which eventually led to the decision to radically revise existing standards using the Evalu-ation and Quality Improvement Program (EQuIP) standards developed by the Australian Council for Healthcare Standards (ACHS) as the main reference. The workshop noted the comprehensiveness of the six functions of EQuIP. Two functions, continuity of care and improving performance, were agreed as priority areas. In subsequent workshops participants devel-oped and prioritised criteria within these two func-tions. The prioritisation was mainly decided by expert-guided consensus. Explicit prioritisation cri-teria could have enhanced the validity of this process. Workshops were the main vehicle to incorporate views of most stakeholders, including the prospective regulator (user of the standards) and the providers who will be assessed. Providers’ views were also incorporated through the preliminary field trial.
Although the desirability of engaging patients is axiomatic, in practice it is problematic.[14] Notably, patients encounter di¤culty in actively contributing to a workshop of a technical nature. Nevertheless, there are recommended alternatives, which should have been considered like small-scale surveys with in-depth interviews, focus group discussion, or rapid appraisals of proxies such as consumer advocates, community leaders, and front-line healthcare practitioners.
A thorough literature review needs to be under-taken to ensure that standards reflect the current best practice based on evidence.[11,15] The incorporation of evidence in this case first took the form of a literature search for standards. This literature search process mainly utilised internet search engines, browsing through the resource collection of CHSM-GMU and consultation with national experts. The search yielded accreditation standards in industrialised countries. The limited institutional access to peer-reviewed journals during development constrained the team from incorporating them. Although some authors recommend that standards should not be developed if the evidence is weak, there has been a growing recognition of the plausibility of drawing on expert opinion in such cases.[5] The issue is how to mix expert opinion with scientific literature in a way that is systematic and rigorous.[16] Formal consensus methods such as Delphi and nominal group techniques based on the RAND* consensus panel are considered to be the best alternatives for this purpose.[5,16,17]
The remaining essential process elements consist of pilot testing, planning for dissemination, implemen-tation, evaluation and revision. There was a record of preliminary field testing conducted in one primary health centre. The final report additionally proposed pilot testing with preceding training on a larger scale. The dissemination process was described to consist of regional workshops and additionally also incorp-orated into the pilot-testing scheme. An operational manual was to be developed to assist implementation and primarily for the accreditation survey process.
In summary, with regard to the development pro-cess, the critical issues are methods to incorporate patients’ views, addressing areas where the evidence is weak and incorporating a plan for continuous revision[4,11]
Attributes of the output
The resulting standards were packaged as a survey instrument encompassing two key functions. Table 1 presents a summary of the standards. The scope does not encompass the whole primary health centres as organisation entities, but rather focuses on their patient care function at the ambulatory clinic. These standards were built on two main principles: con-tinuum of care and continuous improvement. Addi-tionally the standards embraced principles of patient safety and customer satisfaction. Each standard is complemented by criteria to assess compliance. The draft instrument also included guidance for verifica-tion of the standards.
Table 1: Summary of draft accreditation standards for primary healthcare centres in Indonesia
The standards are a combination of structure, process and outcome. Although, there has been con-tinuing debate over the advantages of putting more emphasis on process or outcome, such a balanced combination is seemingly a sensible option.[4,18,19] The criteria accompanying each standard facilitate qual-itative assessment of compliance as a form of perform-ance assessment. The focus of the standards on patient process would also enable development of clinical indicators to complement the qualitative assessment. In relation to this, there was a recommendation to link the standards with the ongoing clinical indicator development project. In line with current recommen-dations, the standards are amenable to performance measurement.[10,11]
Conclusions
Attempts should be made to incorporate patients’ views. Moreover, the standards need to be further consulted for expert opinion through formal methods such as Delphi or the nominal group technique to ensure systematic and rigorous incorporation of evid-ence. Last but not least, a mechanism for regular testing and revision should be planned. Our process can be challenged on methodological grounds. An external evaluation would be needed to address potential biases of such self-assessment. Nevertheless, using limited resources, this practical exercise has drawn attention to key areas for improvement within the complex standard development process. We look forward to the wider use and further development of this critical appraisal approach to healthcare quality assessment.
We would like to thank the Directorate for Primary Care, Directorate-General for Medical Care, Ministry of Health, Republic of Indonesia for their support, and all parties who have contributed to this exercise.
None.
