Clinical Psychiatry Open Access

Keywords

Telepsychiatry; Stigma reduction; Efficacy in depression; Psych deserts

Introduction

In recent years, the advances in medical science have enabled people to live longer, but these advances have been mostly in the area of the physical aspects of health. Extending the lifespan, while a positive trend, has added to the complexity of mental health in many circumstances. As people age, chronic illnesses often worsen, and chronic pain frequently becomes part of a person’s daily life. Other issues also exacerbate the issues of mental health. A person’s retirement and savings might prove to be inadequate to provide for the maintenance of their lifestyle into their extended life. These and many other factors combine to make maintaining a positive mental health difficult. Fifty percent of people living with mental illnesses do not seek professional treatment. This leads to many alternative modalities of treatment that include self-medicating (which is using unsupervised drugs and alcohol) and self-diagnosis (using mediums like WebMD) [1]. Additionally, there are fewer doctors electing to complete psychiatry residency and of the psychiatrists that are practicing, fewer are accepting insurance as payment for their services [2]. One significant advance in mental healthcare over the last decade has involved the delivery of mental health services in alternative modalities. Telemedicine is one of the biggest changes to healthcare providing options for doctors and patients [3]. Telemedicine is the remote delivery of healthcare services, such as health assessments or consultation, over the telecommunications structure. Telemedicine provides convenience, anonymity, lower costs, and easier access to psychiatrists. According to PwC Health Research Institute, patients are willing to use Telehealth service for a wide range of mental health disorders from depression to attempted suicide. Telehealth providers are required to encrypt and transfer the signal between multiple servers to protect the interaction with their patients [4]. Psychiatrists use telehealth in a variety of applications to include office visits, hospitals, prisons, schools, workplace assessments, and in psych-deserts.

Psych deserts are rural areas that do not have mental health care providers. These areas are often forced to use other healthcare specialists like Internal Medicine or Pediatric doctors to provide mental health care. According to a recent report, mental health issues in the workplace [5] especially depression and anxiety, have a significant impact on the US economy with an estimated cost of 1 trillion dollars per year in lost productivity.

Healthcare organizations and systems can take advantage of the aid and assistance virtual platforms offer such as anonymity and freedom from stigma. Among the 50% of individuals with mental illnesses who do not seek treatment of any kind, their perceptions of a stigma associated with receiving a diagnosis of mental illness is among the leading factors influencing their decision to handle their illness on their own. These technological tools can lead to better and increased trust between the organization and the patient [6]. Social determinants have a major impact on people’s healthcare and have been identified as fundamental to increased benefits in delivering healthcare such as better use of health services, involvement in medical decisions, and education about health choices [7]. The new model of successful healthcare delivery includes the provider organizations, patients, and beneficiaries [8]. Healthcare organizations that share knowledge and other assets increase social capital in healthcare when they improve quality of life within a community [9]. As consumers come to demand more control in their healthcare decisions, virtual healthcare will be one of many platforms that will support sound medical and consumer decision making [10]. It is imperative that organizations rise to meet the ever-changing healthcare environment.

Telemedicine is one of the biggest changes to healthcare in the past decade providing options for doctors and patients [3]. Telemedicine is the remote delivery of healthcare services, such as health assessments or consultation, over the telecommunications structure. It provides convenience, anonymity, lower costs to patients, and access to psychiatrists.

According to [1] as many as two thirds of people diagnosed with mental illness do not seek treatment. The internet provides opportunities for healthcare providers to reach patients they would not otherwise be able to reach. This decade has afforded mental health providers the opportunity to deliver low-cost healthcare to patients in a virtual face to face environment, i.e., telehealth [11]. Telehealth is a cost-effective investment that aids in the delivery of mental health services [12].

Literature Review.

Economic trends

There are economic indicators that may contribute to the poor availability of psychiatrists in the United States. Psychiatrists, by the nature of their job, spend more time with their patients than other specialties. Psychiatrists spend on average, 30 minutes to an hour with their patients which means they can see one to two patients per hour whereas internist may see up to seven patients per hour [13]. Patients with comorbid conditions require more time from psychiatrists. Psychiatrists must consider treating the entire patient instead of “from the neck up” symptoms which further reduces the number of patients they treat in a day [14].

Many rural communities do not have psychiatrists and people with mental illness in rural communities are more likely to be hospitalized because they do not seek treatment due to stigma and the proximity to mental health professionals [15]. Patients are more conscious of the need for mental health services and in the past decade there are more intentional collaboration between psychiatrists and internists to combat the overwhelming demand for psychiatric services. In many instances the internists must provide psychiatric services, however limited, to address the mental illness in the patients seen for internal medicine needs. Olfson et al. note that although there are efforts to treat mental illness in primary care settings, managed care organizations disincentivize primary care physicians to treat mental illness with challenging treatment policies that deny payment to physicians for both physical and mental health services on the same day [16].

Technology in healthcare

Psychiatrists of the 21st century have other treatment modalities to compliment pharmacological agents. There has been a shift from curing mental illness to awareness that mental illness can be at best treated through symptom management. As technology continues to improve, mental health providers have the tools needed to not only treat and manage psychiatric diseases but develop a cure [17]. Technology has enabled more efficient means by which providers can deliver better outcomes and build trust with their patients. The internet plays a central role in providing information to patients. “It is also becoming a platform for convening social networks and creating health information” [10].

Telepsychiatry

The concern of a shortage of psychiatrist in the United States has been noted in literature since the early 2000’s. A 2003 Canadian survey of psychiatric trends indicated that by 2018 there would be a shortage of psychiatrist [18]. The need to provide access to psychiatrist, particularly in underserved areas has been an emergent concern. Telemedicine, synonymous with Telehealth and Telepsychiatry are tools that can help increase access to psychiatric services [19]. Telepsychiatry uses electronic communication to provide psychiatric care at a distance and includes telephone, facsimile, electronic mail, Internet and live imaging in a two-way audio-video format [19]. Perednia and Allen [20] define telemedicine as “the use of telecommunication technologies to provide medical information and services”. Telepsychiatry has the capacity and potential to deliver mental healthcare to settings that otherwise would not receive the care due to limited access or self-limiting factors such as stigma [21].

Telepsychiatry is a reliable means of gathering clinical interview data [22]. Many psychiatrists are not comfortable with the technology despite numerous studies that the technology is as efficacious as a traditional face to face visit. Residents trained in Telepsychiatry are well versed to address the mental health challenges impacting communities in the United States [23]. The rapid growth and use of smartphone technology have emerged as a new frontier for Telepsychiatry. Behavioral healthcare services are delivered via apps and internet through smartphones. Smartphone applications can collect pertinent and real time data for clinical use. As smartphones develop and improve camera functionality, smartphones enable psychiatrist the ability to see their patients in real time [21].

Patient satisfaction

Telepsychiatry is well received by patients. In a randomized trial of 140 patients over 24 weeks, Telepsychiatry was as efficacious as face-to-face interactions with psychiatrist in a rural outpatient setting [24]. Brodey, Claypoole, Motto, Arias, Goss [25] carried a study comparing 20 patients seen via Telepsychiatry to 23 patients seen in a traditional face to face setting, determined that patients that were seen via Telepsychiatry were as satisfied as were the patients that were seen face to face. Although the efficacy data is scare on telepsychiatry in various settings, it is generally well received from both patients and doctors [26]. Doctors surveyed in a 2015 comparative study between Telepsychiatry and face to face visits, psychiatrists were well satisfied with treating their patients via Telepsychiatry platforms. Of the psychiatrists surveyed, 62% preferred face to face encounters [27].

Problem statement and research purpose

The primary purpose of this study was to conduct a clinical trial to compare the provision of psychiatric evaluation and medication interventions in a Virtual Out of Office format (VOO) with the provision of psychiatric evaluation and medication interventions in a Virtual in-Office psychiatric encounter (VIO) among adults in Louisiana seeking psychiatric services for the treatment of depression.

Objectives of the study

Objective 1: Describe adults in Louisiana seeking psychiatric services for the treatment of depression on the following selected demographic characteristics.

Objective 2: Describe adults in Louisiana seeking psychiatric services for the treatment of depression prior to receiving treatment and after the completion of treatment on the following psychiatric characteristics: Stigma and Depression.

Objective 3: Compare adults in Louisiana seeking psychiatric services for the treatment of depression who receive a Virtual Out-of-Office (VOO) treatment vehicle with those who receive a Virtual In-Office Treatment (VIO) vehicle on the following psychiatric characteristics: Stigma and Depression.

Materials and Methods

This study is classified as a pretest-posttest control group noninferiority double-blind experimental design diagrammed using configurations from [28] as follows:

R O1 XE O2

R O3 XC O4

The target population was adults presenting with symptoms of Depression that work or reside in the nine parish region known as the Capital Region. The accessible population was adults presenting with symptoms of Depression that work or reside in the nine parish region known as the Capital Region that are seeking treatment from a psychiatric service organization in Louisiana.

The original intent was to have a total sample of 120 participants with 60 in VIO and 60 in VOO treatment groups. However, due to participant withdrawal and other factors of ineligibility, the final sample was 150 total number assigned and 120 eligible participants providing usable data with 75 participants in VIO and 45 participants in VOO. Study participants were randomly assigned to either out of office (VOO) or in office (VIO) treatment groups. Ultimately 75 participants were randomly assigned to receive virtual mental healthcare delivered via in office (VIO) and 45 patients were randomized to receive virtual mental healthcare delivered out of office (VOO). The primary purpose of the study was to compare the provision of treatment in an out of office versus an in office format. Therefore, all treatment services in the study were delivered via a virtual environment to control for the potential extraneous variable of virtual versus face-to-face delivery of treatment. Participants in the VIO group received treatment through virtual delivery but were required to come into the office for treatment.

A form was developed by the researcher to include the following information for each study participant: Coded identifier, age, gender, race, treatment vehicle, employment status, primary diagnosis, and scales to measure stigma and depression. Identity of the participants was anonymous to the researchers.

There were two self-rating scales used in this study: The Depression scale and the Stigma scale. The Depression Scale, a six item self-rating scale, was administered to all study participants diagnosed with Depression according to the Diagnostic and Statistical Manual, 5th ed. (DSM-V) criteria.

The Stigma Scale is a sixteen-item self-rating scale measuring the degree to which stigma is present with the study participant in their work or school environment. Initial contacts were made with the organization’s Clinical Director by the researcher to secure permission to conduct this research study. Meetings were then held with the Case Management Supervisor, and organizational leadership to explain the purpose of the study, discuss the data collection procedures, and to secure support for the project. Approval was also granted by Louisiana State University’s Internal Review Board. Subjects who requested to become a patient at the test location completed intake paperwork and the intake coordinator submitted the packet to a psychiatrist for review and approval. Upon approval the patient was sent to clinical paperwork via DocuSign which included the study instruments. The patient would be reassessed at the 8- week point in their treatment. Patients in the VIO arm were given the instrument while they were on-site and the patients in the VOO arm were given the instrument via DocuSign.

Results

Objective 1

The first objective was to describe study participants on the following demographic characteristics: Race, Age, Gender and Employment status.

Age

The first variable on which study participants were described was their age at the time they entered the study. The mean age of the participants in the study was about 35 (mean 34.6, SD=11.89), and the ages ranged from a low of 18 to a high of 79 years old.

Race

The next variable examined was the race of the participants. The 6 categories consisted of American Indian, Asian, Black, Native American, White, and Hispanic. While the participants were categorized into 6 race groups, 95% of participants were white (n=74, 61.7%) or black (n=40, 33.3%). The other categories collectively consisted of 4.9% (one participant in each of the four categories).

Gender

The next variable examined was the gender of the participants. The majority of the participants (n=91, 75.8%) reported their gender as female and 24% (n=29) of the study participants reported their gender as male.

Employment status

The next variable studied was the study participant’s employment status. The employment categories were Unemployed, Part-Time, Full-Time, Student, Retired, and No Response. The majority (n=71, 59.7%) of study participants indicated that they were employed full-time. Only two (1.7%) of the participants indicated that they were employed part-time (Table 1).

Table 1: Employment status of adult participants who sought virtual psychiatry service for depression.

Objective 2

The second objective was to describe adults in Louisiana seeking psychiatric services for the treatment of depression prior to receiving treatment and after the completion of treatment on the following psychiatric characteristics: Stigma and Depression. The first variable measured was the pre-test measurement for the variable Stigma. Study participants rated their level of agreement on a total of 16 statements. “Strongly agree” was assigned a value of five while “strongly disagree” was assigned a value of one.

To examine the pre-test measurement of perceived Stigma associated with seeking psychiatric treatment among study participants the researcher conducted a factor analysis to determine if underlying constructs existed in the scale. One of the key assumptions underlying the use of factor analysis is that the data is normally distributed. Therefore, the researcher examined the items for degree of deviation from normality using the Shapiro-Wilks test. Additionally, the measure of sampling adequacy (MSA) was examined for both individual items and the overall Stigma scale to determine if the correlation matrix had adequate relationships to justify the use of factor analysis [29]. All data met the assumptions for use of factor analysis. The analysis was conducted as a principal components analysis with a varimax rotation.

To determine the number of factors to be extracted from the scale’s responses, the researcher used a combination of the Latent Root criterion and the scree plot technique. Using the latent root criterion, only factors that explain at least the equivalent of a single variable are extracted [29]. The scree plot technique enables the researcher to identify the optimum number of factors based on the relative proportions of unique variance to common variance among the extracted factors [29]. Using a combination of these criteria the researchers determined that there were two underlying constructs in this scale. Based on the items included in each of the two sub-scales, the sub-scales were labeled as “Intrinsic Factors” and “Extrinsic Factors.”

A pre-test scale score was computed for each participant on each of the two identified underlying constructs. On the “Intrinsic Factors” pre-test sub-scale, the scores ranged from a low of 1.0 to a high of 4·73 (mean=2.16, SD=1.061). On the “Extrinsic Factors” pre-test sub-scale score, the values ranged from 1.00 to 4.60 with a mean of 2.12 (SD=0.767). An overall Stigma pre-test score was then computed as the mean of the responses to the 16 items in the scale. The mean of this overall score was 2.12 (SD=0.925), and the values ranged from a low of 1.00 to a high of 4.25. A scale score was computed for each participant on each of the two identified underlying constructs at the post-test measurement. On the “Intrinsic Factors” posttest sub-scale, the scores ranged from a low of 1.00 to a high of 4.00 (mean=1.78, SD=0.912). On the “Extrinsic Factors” post-test sub-scale score, the values ranged from 1.00 to 4.00 with a mean of 1.89 (SD=0.740).

Depression

The second variable measured was the pre-test measurement for the variable Depression. Study participants rated their level of agreement on a total of 6 statements. “Strongly agree” was assigned a value of five while “strongly disagree” was assigned a value of one. To examine the pre-test measurement on the Depression scale reported by study participants the researcher conducted a factor analysis to determine if underlying constructs existed in the scale. To determine the number of factors to be extracted from the scale’s responses, the researcher used a combination of the Latent Root criterion and the scree plot technique. Using a combination of these criteria the researcher determined that there were no underlying constructs in this scale. An overall pre-test Depression scale score was computed for each participant. The overall Depression pre-test score was computed as the mean of the responses to the 6 items in the scale. The mean of this overall score was 3.01 (SD=1.14), and the values ranged from a low of 1.00 to a high of 5.00. A higher score indicates a higher level of depression. A Depression post-test score was also computed using the responses from the participants taken after the treatment was completed. As with the pre-test score, this score was computed as the mean of the six items in the Depression scale. These Depression post-test scores ranged from a low of 1.00 to a high of 4.83 (mean=2.12, SD=1.04).

Objective 3

The third objective of the study was to compare adults in Louisiana seeking psychiatric services for the treatment of depression on their post-test measurements of the Stigma and Depression by treatment delivery vehicle (VOO and VIO) in the study controlling for corresponding pre-test measurements. Even though individual subjects were randomly assigned to treatment groups which created equivalent groups within the limits of random error, a larger mortality from the VOO group than for the VIO group could have created differences in the groups. Therefore, the Analysis of Covariance test was utilized as an attempt to control for any non-equivalence between the groups. When the treatment vehicle groups were compared on the three post-test Stigma scores (Intrinsic Stigma, Extrinsic Stigma, and Overall Stigma) controlling for the respective pretest Stigma scores, examination of the results revealed that the participants who received the Virtual Out-of-Office (VOO) treatment vehicle had significantly lower scores on all three of the Stigma measures than the group who received the Virtual In- Office treatment vehicle. This indicates that the participants who received their treatment for depression in an out-of-office modality experienced significantly lower levels of Stigma often associated with receiving psychiatric services than did those who received their treatment in an in-office modality (Tables 2-4).

Table 2: Comparison of post-test intrinsic stigma scores by treatment vehicle (VIO and VOO) controlling for pre-test intrinsic stigma scores among adults seeking psychiatric services for the treatment of depression in Louisiana.

^aAdjusted group means: Virtual In-Office (n=48, M=1.92, SE=0.099); Virtual Out-of-Office (n=25, M=1.505, SE=0.0138).

Table 3: Comparison of post-test extrinsic stigma scores by treatment vehicle (VIO and VOO) controlling for pre-test extrinsic stigma scores among adults seeking psychiatric services for the treatment of depression in Louisiana.

^aAdjusted group means: Virtual In-Office (n=48, M=2.001, SE=0.093); Virtual Out-of-Office (n=25, M=1.671, SE=0.0128).

Table 4: Comparison of post-test overall stigma scores by treatment vehicle (VIO and VOO) controlling for pre-test overall stigma scores among adults seeking psychiatric services for the treatment of depression in Louisiana.

^aAdjusted group means: Virtual In-Office (n=48, M=1.948, SE=0.089); Virtual Out-of-Office (n=25, M=1.555, SE=0.0123).

Additionally, the treatment groups were compared on their level of depression as measured by the depression scale. As with the Stigma measures, the Depression pre-test scores were used as a covariate to control for any differences that were created in the treatment groups as a result of experimental mortality. Examination of the analysis revealed that the VOO group exhibited significantly lower Depression scores on the post-test measure (M=1.670, SD=0.920) than was found among the participants in the VIO group (M=2.342, SD=0.919) (Table 5) depression scores among adults seeking psychiatric services for the treatment of depression in Louisiana.

Table 5: Comparison of post-test depression scores by treatment vehicle (VIO and VOO) controlling for pre-test

^aAdjusted group means: Virtual In-Office (n=47, M=1.948, SE=0.134); Virtual Out-of-Office (n=25, M=1.555, SE=0.0184).

Discussion

Virtual healthcare has been changing the treatment options since 2002 across a variety of settings. The technology continues to improve and the comfort level with virtual healthcare has increased both among patients and physicians. The managed care organizations have yet to fully embrace the true potential of this technology and its ability to impact the cost of healthcare. The cost savings come from physicians and health systems treating patients more effectively. People with mental illness have higher healthcare costs and do not maintain the same life expectancy as people without mental illness. People with mental illness often have comorbid conditions that increase the total cost of healthcare. These patients have higher pharmacy costs, higher hospital costs, alternative medical services, and employment costs. All these costs are lower when patients receive mental healthcare.

Reducing the pay for the physician because the patient was seen virtually does nothing but increase profits for the insurance company. The physician still rendered their medical expertise and maintains their cost of healthcare delivery. The idea that virtual medical treatment is cheaper cannot be based on paying physicians less. Doing so will widen the gap of psychiatrist that accept insurance which results in an increased deficit of psychiatrist in an already underserved community. Many psychiatrists in the United States do not accept insurance because the cost of doing business with insurance companies is overbearing.

Conclusion

The Virtual-out-of-office (VOO) treatment level was more effective in reducing the perceived stigma associated with receiving psychiatric treatment than the Virtual-in-Office (VIO) treatment level. The implications of this conclusion lie primarily in the prospect that if the barrier to seeking mental health assistance associated with the perceived Stigma of receiving psychiatric helps can be reduced or even removed, individuals are more likely to seek and receive the help they need. The literature states that 40% of every community is impacted by mental health and that 50% of people do not seek the help they need. When individuals do not seek the mental health they need, it negatively impacts the work environment, family dynamics, and societal safety concerns. Virtual mental healthcare reduces the Stigma associated with seeking mental healthcare and can reasonably reduce this barrier.

The Virtual-out-of-office (VOO) treatment level was more efficacious in treating depression associated with receiving psychiatric treatment than for the Virtual-in-office (VIO) treatment group. Based on this conclusion and the findings of the study, the researcher recommends future depression research which replicates this study to determine if the findings related to the efficacy of virtual treatment are consistent in other geographies. This study was limited to adult patients seeking psychiatric treatment for depression. Future research is needed on depression in teenagers, specifically the impact of virtual psychiatric treatment on career choices and college admissions. Employment Status had an impact on Stigma which may limit the willingness and success of mental health treatment.

Recommendations

The researcher recommends The Centers for Medicare and Medicaid Services (CMS) develops a standardization of credentialing, billing procedure requirements, and the enforcement of the remittance of claims for insurance companies to reduce the burden of administration on healthcare service providers. Currently, medical service providers must adhere to different rules, policies, and procedures with each insurance company associated with seeking reimbursement for claims. The efficacy outcomes and reductions in stigma in this study demand that more should be done to explore the potential to impact the negative effects mental illness has on the United States economy, rising healthcare costs, and the impact to the quality of life. With the declining number of residents entering psychiatry residency programs, the aging of psychiatrist currently in practice, and the prevalence of psychiatrist not accepting insurance, the industry is facing a major problem over the next ten years.

Additional study should be done to determine how doctors can be attracted back to psychiatry residency programs. Medical schools and residency programs should include in their curriculum, instruction on best business practices aimed at enabling doctors to make the best decisions about entering private practice. The need for virtual healthcare options in the United States peaked in March of 2020 with the emergence of the Covid-19 pandemic. The pandemic changed the routines of every healthcare delivery system in the United States. Legislators scrambled to relax antiquated laws to enable healthcare providers to deliver the virtual healthcare services needed amid the crisis caused by the pandemic. The Drug Enforcement Agency (DEA) should remove and or refine the parameters associated with the Ryan Haight Act which requires a face-toface visit with a patient before virtual services can be rendered, especially in mental health. The administration of psychiatric services does not require the psychiatrist to physically touch the patient unlike other specialty medical services.

References

1. Han B, Compton WM, Blanco C, Colpe LJ (2017) Prevalence, treatment and unmet treatment needs of US adults with mental health and substance use disorders. Health Aff 36: 1739-47.
2. Parcesepe AM, Cabassa LJ (2013) Public stigma of mental illness in the United States: A systematic literature review. Adm Policy Ment Health 40: 384-399.
3. Lacktman NM (2016) Five telemedicine trends transforming healthcare in 2016. J Health Care Compli. 43-44.
4. Baker DC, Bufka LF (2011) Preparing for the telehealth world: Navigating legal, regulatory, reimbursement, and ethical issues in an electronic age. Profess Psych Res Prac 42: 405-411.
5. World Health Organization (2019) Mental health in the workplace.
6. Huang KY, Chengalur-Smith I (2014) A social capital perspective to understand individual contribution of social support in healthcare virtual support communities. IEEE: 3489-98.
7. Braveman P, Gottlieb L (2014) The social determinants of health; It’s time to consider the causes of the causes. Public Health Rep (1974) 129: 19-31.
8. Kepros J P, Opreanu RC (2009) A new model for health care delivery. BMC Health Serv Res 9: 1-5.
9. Aizenberg L (2014) Facilitating indigenous women’s community participation in healthcare: A critical review from the social capital theory. Health Socio Rev 23: 91-101.
10. Sarasohn-Kahn J, Barrette J, Barsky E (2008) The wisdom of patients: Health care meets online social media. California Healthcare Found.
11. Barak A, Grohol JM (2011) Current and future trends in internet-supported mental health interventions. J Tech Human Services 29: 155-196.
12. Forster J (2014) Building a healthy future for mental healthcare. British J Healthcare Management 20: 218.
13. Scully J, Wilk J (2003) Selected characteristics and data of psychiatrists in the United States, 2001-2002. Academic Psych 27: 247.
14. Tse J, Chow E, Sultana-Cordero R, Titus-Prescott M, Chiles R, et al. (2011) Motivation-based interventions for obesity in serious mental illness. Psych Annals 41: 473-477.
15. Stewart H, Jameson JP, Curtin L (2015) The relationship between stigma and self-reported willingness to use mental health services among rural and urban older adults. Psychol Services 12: 141-148.
16. Olfson M, Kronenke K, Wang S, Blanco C (2014) Trends in office-based mental health care provided by psychiatrist and primary care physicians. J Clin Psychiatry 75: 247-253.
17. Insel TR, Scolnick EM (2006) Cure therapeutics and strategic prevention: Raising the barfor mental health research. Mol Psychiatry 11: 11.
18. Lin E, Woodside DB, Rhodes A (2003) The canadian psychiatric association practice profile survey: I methods and general sample characteristics. Canadian J Psychiatry 48: 237.
19. Hilty DM, Luo JS, Morache C, Marcelo DA, Nesbitt TS (2002) Telepsychiatry: An overview for psychiatrists. CNS Drugs 16: 527-548.
20. Bashshur R, Reardon T, Shannon G (2000) Telemedicine: A new health care delivery system. Annual Rev Public Health 21: 613.
21. Luxton DD, McCann RA, Bush NE, Mishkind MC, Reger G (2011) mHealth for mental health: Integrating smartphone technology in behavioral healthcare. Professional Psychol: Res Prac 42: 505-512.
22. Monnier J, Knapp R, Grueh BC (2003) Recent advances in telepsychiatry: An updated review. Psych Services 54: 1604-09.
23. Saeed SA, Johnson TL, Bagga M, Glass O (2017) Training residents in the use of telepsychiatry: Review of the literature and a proposed elective. Psych Quarterly 88: 271-281.
24. Cuevas C, Arredondo MT, Cabrera MF, Sulzenbacher H, Meise U(2006) Randomized clinical trial of telepsychiatry through videoconference verus face-to-face conventional psychiatric treatment. Telemed Health 12: 341-351.
25. Brodey BB, Claypoole KH, Motto J, Arias RG, Goss R (2000) Satisfaction of forensic psychiatric patients with remote telepsychiatric evaluation. Psych Services 51: 1305-07
26. Antonacci DJ, Bloch R, Saeed SA, Yildirim Y, Talley J (2008) Empirical evidence on the use and effectiveness of telepsychiatry via videoconferencing: Implications for forensic and correctional psychiatry. Behavioral Sci Law 26: 253.
27. Becevic M, Boren S, Mutrux R, Shah Z, Banerjee S (2015) User satisfaction with telehealth: Study of patients, providers, and coordinators. Health Care Manag 34: 337-349.
28. Campbell DT, Stanley JC, Gage NL (1963) Experimental and quasi-experimental designs for research. Houghton Mifflin Company.
29. Hair JF, Black WC, Babin BJ, Anderson RE, Tatham RL (2006) Multivariate data analysis. New Jersey: Pearson Education, Inc.