Diversity & Equality in Health and Care Open Access

Keywords

Telemedicine; Indicator; Primary health care; Medical teleconsultation; e-Health; Indigenous populations

Key points about what is known

1) The setting-up of health services in the State of Amazonas (Brazil) is a hard task because of geographical and cultural issues including indigenous populations.

2) The classification of medical teleconsultation systems is difficult as well for different patterns reported in literature.

3) Economic studies and randomized control studies are of poor practicability for evaluating the performance of medical teleconsultation systems.

Key points about adds-on

1) The medical teleconsultation system in the State of Amazonas is an attempt to cope with the geographical isolation, lack of expertise and inequality due to cultural diversities.

2) An auditing board of five external experts was set-up to support the evaluation of the performance of the medical teleconsultation of the State of Amazonas.

3) The review of a series of cases drove the board to propose twenty quality indicators focusing on applicants’ and teleconsultants’ performances and on the flow of proceedings.

4) The quality indicators focusing on applicants’ performance suggest the need of improvement through formal courses.

Introduction

Nearly all of the many experiences in teleconsultation, reported over recent years, focus on either specific disciplines, such as cardiology, dermatology or technologies, such as real-time ultrasound scan or early detection and follow-up of diseases (e.g. epilepsy) [1-3]. Thus, any such experiences deal with selected health subjects and involve a few specialized researchers. Although Information and Communication Technologies really provide an added value in terms of accuracy, speed and expert qualification, such experiences seem to be non-essential in providing health, especially in the urban areas where patients find alternatives to solving their health problems. Actually, such experiences are often designed as pilot projects and therefore they are not integrated into the health system as a service. Thus, their impact on the overall health of the population remains poor or unproven [4,5]. The only model of health system integrated medical teleconsultation is the support to the Primary Health Care doctors (PCD), living in remote areas, with no expertise available at short distances, in instances where they encounter cases of medium complexity beyond their skills [6]. Such models of teleconsultation systems demonstrably reduce the patient’s medical error vulnerability [7]. The medical teleconsultation system of Amazonas fits this profile.

Located in northern Brazil, the state of Amazonas covers 1,559,149 km². The total population of the state is estimated at 3,938,336 inhabitants. Around 2 million people live in the municipal area of the capital, Manaus, with the remaining 1.8 million inhabitants spread over the other 61 municipalities lying within the Amazon Rainforest. Thus, the mean surface of the municipalities is very wide (25,373 km²) and the density of the population very low (2.2/ km²) [8].

In the navigable portions of the Amazon River Basin, the transporting of people and goods usually occurs by boat; however, the low density of the population means many communities remain nevertheless isolated.

Of the mobile telephone companies operating in the state of Amazonas, only one has satisfactory repeater coverage. However, the wireless repeater cell phone signal is limited to the main settlement areas of the municipalities, with Internet data transmission being often not supported and the loss of signal common during thunderstorms. Transferring data is erratic and restricted to the use of satellite antenna.

In addition, in the state of Amazonas 168,680 inhabitants are self-declared as indigenous, representing the greatest concentration (20%) of the entire indigenous population in Brazil [9]. They reportedly represent 62 ethnic groups speaking 45 languages [10].

Many reports suggest that such ethnic minorities get access to the health services to a much lesser extent [11,12]. The quality of such services is low, but with the expansion of Information and Communication Technologies there is the promise of improvements [13,14]. As such, the geographic and cultural ethnical context of the state of Amazonas is a challenge for any social services to work and often demands personalized solutions.

Nevertheless, the unified and universal Brazilian national health system (Sistema Único de Saúde – SUS) is able to provide Primary Healthcare (PHC) through family healthcare teams, led by a general practitioner (Primary HealthCare Doctor - PCD), in health posts. Moreover, most of the main towns within the municipalities have their own district hospital where Secondary HealthCare is provided. Medium complexity diseases and lifethreatening cases are referred to such hospitals. However, when dealing with urgent cases or elective surgery of the highest complexity of care, the patient is evacuated to Manaus, via expensive air transport [15]. In the case of indigenous patients and families, there is frequent resistance to such referrals, with patients even opting instead to employ traditional medicine, creating tension with medical authorities [16].

Support for coping with isolation and lack of health expertise is provided through the Telemedicine and Telehealth Center (TTC). The TTC provides asynchronous (e-mail based) and synchronous (teleconferencing) medical teleconsultation service via Internet satellite connection through a portal of the Ministry of Health [17]. PCDs registered through the portal, practicing in the countryside of Amazonas, can request consultation on their occurring cases by logging into the portal and posting the case asynchronously. Afterwards, TTC relays the cases to the relevant consultants, who return their counseling to the applicants.

All together, the geographical and infrastructural setting makes this teleconsultation system a universally unique challenge. Moreover, although in the literature many of the experiences in telemedicine places stress on workloads, papers focusing on auditing and quality indicators of telemedicine systems are few [18].

Thus, the purpose of this observational study is to establish the basis for teleconsultation system auditing by instituting and estimating quality indicators (QI), while profiling the actual demand in terms of applications from January to September 2016.

Materials and Methods

Clinical case posting and acceptance

In the municipalities, a technician in a telemedicine center manages the hardware and satellite Internet connections. The health professionals (PCDs and, occasionally, health assistants and nurses) subscribe to an account in the portal. All of them work as PHC professionals in health posts, which serve as the patient entry point into the health system, providing care for whatever type of pathology the individual may have, with the exception of emergency conditions. The process of medical teleconsultation (Figure 1).

Figure 1: The process of medical teleconsultation.

Whenever the PCD attends a patient (step 1), he usually solves low complexity problems on his own with no need of external support. He may also come across cases of medium complexity, mono- or multidisciplinary, exceeding the common knowledge of PHC professionals. In such cases, he can log into the portal and request asynchronous medical teleconsultation (step 2). According to TTC issued guidelines, he inputs in free text form: a) the full identification data of the patient (name, surname, gender, date of birth, municipality of residence, weight and height); b) the clinical history related to the actual reason for the consultation; c) the relevant semiotic aspects of the physical examination; d) the hypothesized diagnosis or the differential diagnosis diagram; e) on-going and relevant previous medications and f) the specific demands or questions he wishes to be addressed by the consultants. The free text application may have attachments such as pictures of important semiotic features (e.g. lesions of the skin, of goiter, etc.) to show, X-ray tests, electrocardiograms, growth cards, and other significant and pertinent clinical documents in the most common electronic formats (JPEG, PDF, Word, Mp4, etc.).

An internal medicine clinician (Medical Coordinator - MC) goes through the information provided by the application asynchronously as well, within a maximum delay of 8 hours’ time. Afterwards, the MC classifies the application according to the Second Edition of the International Classification of Primary Care [19]. The MC may reject the applications under four conditions: 1) Requests unfeasible for ethical or procedural reasons; 2) Low complexity of care that ought to be solved at PHC level in the municipality; 3) No compatibility with the capacity of medical teleconsultation to address (surgery, complex procedures of cancer staging, psychological and psychiatric complicated evaluations, etc.) or 4) Application not drawn up according to the guidelines issued. The MC justifies the reasons for rejection and gives guidance to the PCD to improve the quality of his future applications.

The process of teleconsultation

If the application is accepted with only some minor information missing, the MC returns the contact with the PCD synchronously (teleconferencing through various voice-over Internet protocols, telephone calls, according to the PCD’s available communication tools) (step 3). In this manner, any unclear issue in the application is ascertained. The MC is required to be sympathetic to the imperfections and incompleteness of the applications given that, basically, such applications deal with clinical cases of medium complexity that are beyond the skills of the PCD to solve.

In the next step, the MC, as an internal medicine clinician, takes the responsibility for developing the strategy for solving the case. For this purpose, he may contact (asynchronously or, more often, synchronously) one or more experts whose locations are within three continents (Americas, Europe, Africa) (step 4). Experts of various medical disciplines are included in a database of over 90 experts. Previously, all of them, of proven skills and experience, had declared their commitment to advising the MC and have obviously gained the MC’s trust.

By means of such a teleconsultation, the MC builds the strategy of clinical problem solving by merging the required knowledge, establishing priorities, and drawing up a plan of action, in accordance with the principles of the internal medicine holistic approach, the common worldwide shared official medical deontology, and the feasibility of the medical procedures of the Brazilian health system (step 5) [20]. From this point, three possibilities may occur: 1) the patient’s problem can be solved by the PCD through simple advice (ex: prescribing and adjusting treatments for a resistant chronic heart failure or for a specialized disease such as eczema); 2) the PCD can solve the problem through a procedure he has never done before and for which he has to be quickly trained (ex: carrying out an ultrasound scan to diagnose gallstones in the gallbladder) or 3) the patient unpredictably has to undergo procedures or treatments that no health professional in the municipality can perform (ex: surgery of the highest complexity).

Whichever the occurrence, the MC chooses the smoothest and safest way to proceed. He may either reply to the PCD posted application on his own or, more often, refer the case to one of the teleconsultants (TC) registered in the platform (step 6) and, afterwards, this selected TC replies to the applicant through the portal (step 7). The quality of such counseling is much related to links, evidences, multimedia tutorials, and recorded PowerPoint presentations, which can be attached to the report by the MC and the TC [21].

The case is concluded and filed through a monitoring process, including the evaluation of the applicant’s satisfaction regarding the counseling received (step 8).

The classification of clinical cases according to the problem solving strategy

The established problem solving strategy drives on toward further classification of the case. While going over the application, the MC has to make decision regarding: a) urgency (except emergencies, which cannot possibly be solved through teleconsultation) and b) the complexity of competencies required and therefore the number and profile of the consultants to call in.

The definition of urgency is based on the usual reaction time of the asynchronous medical teleconsultation system available. Whenever the MC determines that the asynchronous teleconsultation may be not an effective and quick tool, he puts in for a synchronous teleconsultation as soon as possible. During that meeting, all three (PCD, MC, TC) make all decisions and take action accordingly.

With regards to the complexity of the teleconsultation, it is divided into two groups: a) mono-disciplinary and b) multidisciplinary. The requests for mono-disciplinary case solvable teleconsultations are usually included in the following domains: a) dermatology; b) odontostomatology; c) neuro-psychiatrics (including infant and child development) and d) obstetricsgynaecology. The multi-disciplinary cases may be of the most varied pattern of complexity and, therefore, we try to merge these into the minimum number of categories and thus reduce the fragmentation into specialties. To simplify matters, we group them into: a) internal medicine (also including pediatric and newborn cases) or b) traumatology (including surgery in broad perspective).

Mono-disciplinary cases are usually directed by the MC to the relevant TC and therefore involve a single operation of virtual referral. Afterwards, the consultant advises the PCD of his own accord.

Multi-disciplinary cases are managed under the MC’s direct coordination. The MC collects opinions from several consultants and finally advises the PCD.

QI of medical teleconsultation

Despite the efforts of the American Telemedicine Association, the literature still provides poor and generic guidance as to the quality indicators of multi-disciplinary telemedicine services, whose reports of activities remain without any valuable audit [22,23]. Thus, an audit board was set-up. It was composed of 5 TTC-external international specialists profiled as follows: a) one internal medicine specialist, professor in clinical medicine and clinical methodology; b) one expert in public health system management from an agency of the United Nations; c) one PHC medical doctor (General Practitioner), experienced in telemedicine network management; d) one dermatologist with a background of teleconsultation and e) one manager of the regional clinical laboratories network.

The tasks of such a board were: a) to check the repeatability of proceedings of medical teleconsultation; b) to establish the QI and the process monitoring parameters of the teleconsultation system; c) to review the series of the applications according to the selected QI and parameters of evaluation and estimate the percentage of occurrences of the QI and d) to the extent possible, to establish desirable standards of the QI.

The board verified the whole series of cases occurring over the period January 2016 to September 2016, when the composition of the TTC team, the set-up of Internet access points in the municipalities, and the network of applying PCD remained steady. In the whole process of the evaluation, the members of the board got no access to the identity and race of the patients.

20 QI were selected as relevant and applicable in any teleconsultation model addressing the support to PCD.

Of these, 17 were measurable quantitatively, albeit with some restrictions. The remaining 3 could be estimated only semiquantitatively in one of the three categories: 1) satisfactory; 2) improving or 3) unsatisfactory.

The QI were numbered progressively according to the sequential steps of the teleconsultation procedure. The QI 1-6 were dependent upon the applicant’s skills and commitment. However, owing to the poor applicability of the method to any kind of discipline, some evaluations were carried out only in dermatological cases, in that these made up a homogeneous, as well as the most numerous, case series.

The QI 7-15 were dependent only upon the TTC’s capacities. The remaining QI 16-20 were related to the intrinsic methodology of the Amazonas teleconsultation system.

Quality Indicators of the ratio of cost effectiveness of the medical teleconsultation system of Amazonas, though of interest, could not be calculated within the scope of this study, as the TTC budget includes expenses for various other activities beyond medical teleconsultation, namely tele-education, virtual seminars, formal courses for students from the Faculties of Medicine , Nursing, and Odontology. Considering that the TTC staff is normally engaged in these activities, the budget items could not be effectively broken down to individual activities.

Given the lack of references and guidance from literature, the desirable standards of the QI were flexibly and arbitrarily established by the board, according to other telemedicine system management experiences.

Results

Workload

72 health professionals from 22 municipalities applied for teleconsultation over 9 months, as shown in Figure 2. Five of these municipalities (São Gabriel da Cachoeira, São Paulo de Olivença, Tabatinga, Santa Izabel do Rio Negro, and Barcelos) are among the 10 municipalities in Brazil with the maximum number of rural indigenous people, while Manaus is in the Top 10 of municipalities with the greatest proportion of urban indigenous population [24] (Figure 2).

Figure 2: Different municipalities applied for teleconsultation.

The total amount of applications was 346. The municipality of Canutama presented the minimum (only 1 application), while Tabatinga stood at the maximum (149 applications).

The male/female ratio of the patients was 0.8.

The mean age of the patients was 31 + 27. For the female sex the mean age was 32 + 26. For the male sex the mean age was 31 + 27.

281 (81%) applications were attended. The remaining 65 (19%) were rejected.

The attended applications were grouped as follows: 115 internal medicine (41%); 104 dermatology (37%); 31 neuro-psychiatric (11%); 13 obstetrics-gynecology (5%), 11 traumatology (4%) and 7 odontostomatology (2%).

Of the total applications, 61 (18%) were held and forwarded as urgent by the MC. The disciplines of the urgent cases were: 40 internal medicine (66%), 10 traumatology (16%), 7 neuropsychiatrics (12%) and 4 obstetrics-gynecology (6%).

The QI

The 20 QI selected and their estimation over the case series is summarized in the table. In the last column desirable standards are reported and commented (Table 1). QI 16 is noteworthy. It considers the cases which could not be solved or even supported by teleconsultation as a method (cold cases). Of the 12 cases (8 classified as internal medicine and 4 as neuro-psychiatric): a) 6 were of a multi-disciplinary complexity that cannot be integrated and coordinated under the internal medicine; b) 3 involved the execution of surgical procedures and were therefore off-limits for teleconsultation and c) 3 presented no chances of collecting even PHC basic information and physical examination signs, for the impossibility to trace back indigenous patients for their nomadism.

Table 1: Quality indicators of the teleconsultation system of Amazonas.

Discussion

The multi-disciplinary teleconsultation systems usually lay within an ecological niche carved in a vacuum of the chain of the health services. Over the course, they are continuously reshaped, according to the sudden alterations in the environment (new investments in health, turnover of health managers and human resources for health in the field, new connectivity and transport opportunities etc) and therefore to the changing demand. This is why every teleconsultation system has its own profile and is not necessarily duplicable elsewhere [32]. Recommendations and guidelines issued about evaluation often turn out to be generic and poorly applicable [33].

In this respect, Tuerk and coll. published a detailed review to stress the difficulty in carrying out randomized-control studies on the evaluation of telemedicine systems, which are held by some authors as golden standard for impact [34]. Tuerk considers double-blind studies as nearly impossible and raises ethical concerns [35].

Krupinski and Bernard support the idea that observational case series studies, as ours is, are still the most supportive [36].

However, comparing telemedicine system performance through workloads with no hint or comment about the health system background makes really poor sense, albeit sometimes helpful in teleconsultation system management [37].

Economic analyses studies are also puzzling. Bergmo arguments whether they can be trusted and Mistry states that there is no conclusive evidence that telemedicine interventions are cost-effective compared to conventional health care [38,39]. But in Amazonas conventional health care is really not competitive, even not sustainable [40].

However, given how difficult it may be, establishing the QI is a mandatory task to help the teleconsultation system managers in monitoring the performance, understanding the changeable environment and demands, making quick adaptive decisions and maintaining the benefit-cost ratio.

Establishing “normal” ranges of QI presented is another difficult task. However, among optimum, attainable, and desirable, we believe that, nowadays, desirable is the only practical choice. Currently, the optimum is too easy and perhaps even unrealistic, while the attainable still remains unpredictable.

Focusing on the QI of the applicant’s performance, the poor quality of anamnesis, physical examination, and attached pictures demands a strategy of permanent refresher training of the PCD regarding good practices. A TTC must mandatorily operate with both teleconsultation and tele-education for its improved performance.

Among the QI of TTC’s performance we believe that the large size of available expert database (n. 7) and the reaction time (n. 10) are the most noteworthy. The expert database must be as big and as international as possible to escape selfreferencing of small schools of medicine and therefore to increase accountability and auditability. Moreover, the trust of the PCDs is strictly reaction-time dependent.

Among the method-dependent QI, item n. 20 represents the most disturbing shortcoming. A system of capturing and monitoring information on resources, services, and facilities in the field is an indispensable tool.

Additionally, checking the repeatability of proceedings by periodically auditing through external experts is to be developed.

Conclusion

Although not yet definitive, the proposed set of QI of performance in Amazonas is a practical attempt to move forward in the evaluation of teleconsultation systems. Unfortunately, we could include no cost-effectiveness indicator.

Further observational case series studies are needed just for performance monitoring purposes. An external and independent audit board should always support such studies.

Furthermore, concrete data on indigenous access to teleconsultation should be obtained to properly measure the full reach of this essential healthcare tool in the Amazon.

Ethical Approval

This study was carried out conforming the Declaration of Helsinki and under the permission of the telemedicine platform managing authorities of the Ministry of Health of Brazil. In particular, as it is also informed in the paper, the involved investigators could not have access to the identity and race of the patients throughout the revision of the case series. The methodology of the research preserved anonymity and privacy of the patients at full extent.

Acknowledgement

The authors acknowledge the managing work performed by Prof. Cleinaldo Almeida Costa, Chancellor of Amazonas State University and former Director of the Telemedicine and Telehealth Center.

Source of Funding

No grant was given for this research by any sponsor. Authors and external experts worked on pure voluntarism and gratuity, just aiming to give a scientific contribution to this controversial issue.

Conflict of Interest

The authors declare that none of them have any conflict of interest in this research.

References

1. Chen RH, Snorrason M, Enger SM (2016) Validation of a Skin-Lesion Image-Matching Algorithm Based on Computer Vision Technology. Telemed J e-Health. 22: 45-50.
2. Frumento E, Colombo C, Borghi G (2009) Assessment and analysis of territorial experiences in digital tele-echography. Ann Ist Sup Sanita. 45: 363-371.
3. Velasquez SE, Chaves-Carballo E, Nelson EL (2016) Pediatric Teleneurology: A Model of Epilepsy Care for Rural Populations. Pediatr Neurol. 64: 32-37.
4. Caffery LJ, Farjian M, Smith AC (2016) Telehealth interventions for reducing waiting lists and waiting times for specialist outpatient services: A scoping review. J Telemed Telecare. 22: 504-512.
5. Douglas MD, Xu J, Heggs A (2016) Assessing Telemedicine Utilization by Using Medicaid Claims Data. Psychiatr Serv 68: 173-178.
6. Kolsoum D, Kambiz B, Seyed MT (2016) Teleconsultation and Clinical Decision Making: a Systematic Review. Acta Inform Med. 24: 286-292.
7. Campanella N, Morosini P, Sampaolo G et al (2015) Medical teleconsultation to general practitioners reduces the medical error vulnerability of internal medicine patients. Eur J Int Med. 26:675-679.
8. Instituto Brasileiro de Geografia e Estatística (n.d).
9. Instituto Brasileiro de Geografia e Estatística (2012) Os indígenas no Censo Demográfico. Primeiras considerações com base no quesito cor ou raça.
10. Instituto Socioambiental (2010) Povos Indígenas do Brasil.
11. Guimarães AS, Mantovani SA, Oliart-Guzmán H (2016) Prenatal care and childbirth assistance in Amazonian women before and after the Pacific Highway Construction (2003-2011): a cross-sectional study. BMC Womens Health 16: 37.
12. Aith Fernando (2008) Saúde indígena no Brasil: atual quadro jurídico-administrativo do estado brasileiro e desafios para a garantia do direito à saúde da população indígena. Revista de Direito Sanitário. 9: 115-132.
13. Benzaken AS, Bazzo ML, Galban E (2014) External quality assurance with dried tube specimens (DTS) for point-of-care syphilis and HIV tests: experience in an indigenous populations screening program in the Brazilian Amazon. Sex Transm Infect. 90: 14-18.
14. Miller DP, Weaver KE, Case LD (2017) Usability of a Novel Mobile Health iPad App by Vulnerable Populations. JMIR 5: e43.
15. Pan-American Health Organization (2013) Teleconsultoria e telessaúde na atenção básica.
16. Gadelha Luciana Fernandes Portal Lima (2009) A Articulação dos Sistemas Tradicionais e Clássicos de Medicina na Atenção à Saúde dos Povos Indígenas - Uma Análise da Atuação do Ministério Público Federal no Caso da Criança Indígena da Etnia Tukano - Alto Rio Negro.
17. Portal of telemedicine and telehealth of the Ministry of Health of Brazil (n.d).
18. Latifi R, Gunn JK, Bakiu E (2016) Access to Specialized Care Through Telemedicine in Limited-Resource Country: Initial 1,065 Teleconsultations in Albania. Telemed J e-Health 22: 1024-1031.
19. World Organization of National Colleges, Academies and Academic Associations of General Practitioners/Family Physicians - WONCA (n.d). International Classification of Primary Care (2nd Edition
20. Athanasiou T, Darzi A (2011) Evidence synthesis in healthcare. A practical handbook for clinicians Springer.
21. Campanella N, Giovagnoli M, Lucci R (2008) Ver para Comprendeer. Tecnologias Multimédia Para Todos. Offset publication. Ancona.
22. American Telemedicine Association (2007a) Core standards for telemedicine operations.
23. Bernocchi P, Scalvini S, Tridico C (2012) Healthcare continuity from hospital to territory in Lombardy: TELEMACO project. Am J Manag Care. 18: e101-108.
24. Instituto Brasileiro Geográfico e Estatística (n.d). Indígenas: Gráficos e Tabelas (n.d).
25. American Telemedicine Association (2007b) Practical Guidelines for Teledermatology. Available at: https://dev.americantelemed.org/docs/default-source/standards/practice-guidelines-for-teledermatology.pdf?sfvrsn=4 Date last accessed 05/24/17.
26. Rugarli C (2011). L’ABC del metodo clinico. Elsevier, Springer-Verlag London limited editors.
27. Wenger NS and Young RT (2007) Quality indicators for continuity and coordination of care in vulnerable elders. J Am Geriatr Soc 55: S285-S292.
28. Campanella N, Novelli S, Morosini PP et al (2017) Use of the citizen’s electronic health record by doctors and patients in a mature application throughout a district of the national health system of Italy. Eur J Int Med. 37: e47–e49.
29. Campanella N, Claudio de Souza (2014) Tecnologias de Informação e Comunicação Desafiam a Ética Médica. Multimedia 9’ 34”, Available at: https://youtu.be/O8tbfLIDqRE Date last accessed 06/15/17.
30. Campanella N, Francioni O, Taus M (2004) When And How Is Medical Teleconsultation To Doctors Practising In Remote Areas Of Developing Countries Convenient And Reliable? About Eight Clinical Cases. Rec Prog Med. 1: 5-10.
31. Stern S, Cifu A, Altkorn D (2015) Symptom to Diagnosis. An evidence-based guide. Mc Graw Hill Editor. International Editions ISBN 978-1-25-925253-2.
32. Bashshur RL, Shannon G, Krupinski EA (2013) Sustaining and realizing the promise of telemedicine. Telemed J e-Health. 19: 339-345.
33. Department of Health and Human Services (2013). Assessment of Approaches to Evaluating Telemedicine. Available AT: https://aspe.hhs.gov/health/reports/AAET/aaet.htm/ Date last accessed 06/15/17.
34. Davalos ME, French MT, Burdick AE (2009) Economic evaluation of telemedicine: Review of the literature and research guidelines for benefit-cost analysis. Telemed J e-Health 15: 933-948.
35. Tuerk PW, Fortney J, Bosworth HB (2010) Toward the development of national telehealth services: The role of Veterans Health Administration and future directions for research. Telemed J e-Health 16: 115-118.
36. Krupinski EA, Bernard J (2014) Review: Standards and Guidelines in Telemedicine and Telehealth. Healthcare 2: 74-93.
37. Marcolino MS, Figueira RM, Dos Santos JP et al (2016) The Experience of a Sustainable Large Scale Brazilian Telehealth Network. Telemed J e-Health 22: 899-908.
38. Bergmo T (2009). Can economic evaluation in telemedicine be trusted? A systematic review of the ?literature. Cost Eff Resour Alloc, 7.
39. Mistry H (2012) Systematic review of studies of the cost-effectiveness of telemedicine and telecare. ?Changes in the economic evidence over twenty years. J Telemed Telecare 18: 1-6.
40. Dekio I, Hanada E, Chinuki Y (2010) Usefulness and economic evaluation of ADSL-based live interactive teledermatology in areas with shortage of dermatologists. Int J Dermatol 49: 1272-1527.