OUP user menu

Development, Validation, and Use of English and Spanish Versions of the Telemedicine Satisfaction and Usefulness Questionnaire

Suzanne Bakken , Lorena Grullon-Figueroa , Roberto Izquierdo , Nam-Ju Lee , Philip Morin , Walter Palmas , Jeanne Teresi , Ruth S. Weinstock , Steven Shea , Justin Starren
DOI: http://dx.doi.org/10.1197/jamia.M2146 660-667 First published online: 1 November 2006

Abstract

Objectives: To describe the development and validation of low literacy English and Spanish versions of the 26-item Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ), report telemedicine satisfaction and usefulness ratings of urban and rural participants in the Informatics for Diabetes Education and Telemedicine (IDEATel) project, and explore relationships between utilization and perceptions of satisfaction and usefulness.

Methods: Data sources included TSUQ, utilization data from IDEATel log files, and sociodemographic data from the annual IDEATel interview. Psychometric analyses were conducted to examine the reliability and validity of TSUQ. Data were analyzed using descriptive, correlational techniques.

Results: The principal components factor analysis extracted two factors (Video Visits, alpha=.96, and Use and Impact, alpha=.92) that explained 63.6% of the variance in TSUQ satisfaction scores. All satisfaction and usefulness items had mean scores of greater than 4 on a 5-point scale. Those from urban areas reported significantly higher ratings on both factors than rural participants as did those who did not know how to use a computer at baseline. Mean frequency of utilization of IDEATel components was highest for blood sugar testing followed by web site for reviewing results, blood pressure testing, video visits, and ADA educational Web pages. Associations between utilization and perceptions of satisfaction and usefulness varied among IDEATel components.

Conclusion: Psychometric analyses support the construct validity and internal consistency reliability of TSUQ, which is available in both English and Spanish at a readability level of 8th grade. Both rural and urban participants reported high levels of satisfaction and found all IDEATel components useful. Further work is needed to examine the relationships between utilization and perceptions of satisfaction and usefulness and to explore the effects of location (urban versus rural) and ethnicity on satisfaction with telemedicine services.

Introduction

Acceptance and use of telemedicine applications is a prerequisite to reaping the potential clinical benefits of the technology. Consequently, it is important to complement studies of telemedicine effectiveness with studies that examine perceptions of satisfaction and usefulness as well as actual utilization of various telemedicine services. In this paper, we report the development and validation of low literacy English and Spanish versions of the Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ) and their use in the Informatics for Diabetes Education and Telemedicine (IDEATel) project.

IDEATel is a randomized controlled trial (RCT) that is comparing telemedicine case management with usual care in older patients with diabetes mellitus in New York State.13 Participants randomized to the intervention group received a Web-enabled home telemedicine unit (American Telecare, Inc.; Eden Prairie, MN) with modem connection to an existing telephone line. The home telemedicine unit consisted of four components: 1) video camera and microphone for conferencing with nurse case managers; 2) home glucose monitoring device (One Touch Sure Step; Lifescan, Inc.; Milpitas, CA) and blood pressure cuff (UA-767 Blood Pressure Monitor; A & D Medical; Milpitas, CA) for uploading of fingerstick glucose and blood pressure readings into the clinical database; 3) access to patient clinical data; and 4) access to an educational Web site created by the American Diabetes Association in regular and low literacy versions in English and Spanish. Participants were assigned to a nurse case manager who, under the supervision of a diabetologist, interacted with patients using the home telemedicine unit and case management software. At one year of follow-up, telemedicine as compared to usual care demonstrated improved glycemic control, blood pressure levels, and total and LDL cholesterol levels.4 The magnitude of intervention effects was similar for urban and rural participants.

Effectiveness studies are increasing in number, however, to date, patient satisfaction has been the primary focus of patient-oriented evaluations of telemedicine.5,6 Two recent systematic reviews of telemedicine satisfaction studies suggest that although the findings of most studies indicate a high level of patient satisfaction, a number of methodological concerns limit the generalizability of the individual studies and present barriers to knowledge development.7,8 These methodological concerns include: 1) design issues such as lack of control groups; 2) small sample sizes, typically less than 100; 3) measurement issues including lack of detail on survey reliability and validity and inclusion of multiple constructs in a single item; 4) inadequate attention to communication between patient and provider; 5) response bias and loyalty (i.e., patients tend to positively evaluate healthcare even in the absence of using some services); and 6) selection bias and response rates. Similar to patient satisfaction studies in general, telemedicine satisfaction studies also frequently collapse satisfied and very satisfied in the data analysis and may consequently fail to capture important nuances between the two constructs.9 Also of note, most telemedicine satisfaction studies included in recent systematic reviews focus only on interactive video conferencing rather than on a broader suite of telemedicine services. In addition, little is known about variance in satisfaction across medical specialty or mode (e.g., interactive videoconferencing, store and forward technology) of the telemedicine application.8 Whitten and Mair also suggest that research that focuses on specific questions of interest rather than generic satisfaction is required for knowledge development in the area of telemedicine.10

Studies specifically conducted in the clinical domain of diabetes mellitus have similar methodological concerns to other telemedicine satisfaction studies. Ferrer-Roca et al. reported overall satisfaction with continuity and self-efficacy of care in 12 patients who used a Web-based diabetes management system every two days on average, but provided few details regarding the development and validation of the online questionnaire.11 In a small RCT (n = 56), Izquierdo and colleagues demonstrated improvements over time in HbA1c and Problem Areas in Diabetes scale scores for patients who received diabetes education through telemedicine or in person, but no differences were found between the groups.12 They also reported that patient satisfaction was high in the telemedicine group. In a RCT that demonstrated cost and time savings using telecare as compared to usual care for managing intensified insulin therapy, 85% of 27 telecare patients judged it as superior to conventional care for reasons such as better blood glucose surveillance, faster interventions, and no travel time.13 No information was provided about the patient questionnaire that was the basis of the judgment. Yip and associates reported the development of the Telemedicine Satisfaction Questionnaire in a sample of 38 patients with diabetes mellitus who participated in four interactive tele-education group sessions.14 In contrast to most telemedicine satisfaction studies, the authors provided data supporting the internal consistency reliability, construct validity, and predictive validity of the questionnaire. However, the generalizability is limited by the small sample size.

In this paper, we describe the development and validation of English and Spanish versions of the 26-item TSUQ, report telemedicine satisfaction and usefulness ratings of urban and rural participants in the IDEATel project, and explore relationships between utilization and perceptions of satisfaction and usefulness. The evaluation of perceptions of satisfaction with and usefulness of telemedicine in a sample of urban and rural telemedicine participants addresses a number of the significant methodological concerns reported in the literature including: inadequate sample size, measurement issues, lack of content related to patient-provider communication, selection bias and response rates, and limited scope of telemedicine services evaluated.

Methods

Study Design

We used a descriptive, correlational design to answer three research questions:

  • 1) What are the psychometric properties of the TSUQ?

  • 2) What are the telemedicine participants' perceptions of satisfaction with and usefulness of IDEATel?

  • 3) What are the relationships between utilization of IDEATel and perceptions of satisfaction and usefulness?

Sample

The study sample comprised active enrollees in the intervention arm of the IDEATel study. Criteria for inclusion in RCT were: 55 years or older, Medicare beneficiary, physician's diagnosis of diabetes mellitus, receiving treatment for diabetes (diet, oral hypoglycemic agent, or insulin), residence in a federally-designated medically underserved area in New York State, and fluency in English or Spanish. Exclusion criteria were moderate or severe cognitive, visual, or physical impairment, or the presence of severe comorbid disease.

Study Measures

Sample characteristics. Sample characteristics were collected through interviews at enrollment and annually. These included items related to ethnicity, age, health beliefs, and reason for enrollment in IDEATel. Diabetes mellitus–related items were: whether or not the participant took insulin, frequency of glucose fingersticks, and perceived quality of diabetes care during the last 12 months.

Perceptions of satisfaction and usefulness. We developed a 26-item low literacy survey instrument in English and Spanish to measure telemedicine participants' perceptions of satisfaction and usefulness (Table 1), the TSUQ. Our goals for the instrument development process were to create or adapt items that addressed specific IDEATel components as well as overall satisfaction with the telemedicine process; achieve a readability level no greater than 8th grade; and implement in English and Spanish. We used the following steps to achieve these goals:

  • 1) We selected 51 items from a variety of sources including the Telemedicine Perception Questionnaire (TMPQ),1517 the American Telemedicine Association Home Technology Survey Item Bank,18 blood pressure and blood glucose self-testing and monitoring items, general satisfaction items,19 and the Engagement with Healthcare Provider survey.20

  • 2) Four telemedicine experts prioritized items for inclusion in the survey and suggested new items.

  • 3) We edited the reduced set of items to create a common structure and response scale and to lower the readability level.

  • 4) Two IDEATel research team members familiar with the data collection process reviewed the items and recommended changes to the English version to prevent awkwardness in the Spanish version (e.g., editing of negatively worded items so that a higher rating always meant a positive rating, changing of a ranking task to a rating task).

  • 5) A bilingual research team member translated TSUQ into Spanish and the translation was subsequently approved by the Hispanic Recruitment and Retention Center at Columbia University Medical Center.

The resulting TSUQ included 26 items. Twenty-one items focused on perceived satisfaction with a Likert-type rating scale of strongly disagree to strongly agree. Five items were related to the perceived usefulness of the five IDEATel components with a Likert-type rating scale of not at all useful to very useful. The mean readability of the 26 items was 8th grade. Readability scores of the individual items ranged from 2.2 to 12.0 and are shown in Table 1.

View this table:
Table 1

Perceptions of Satisfaction and Usefulness Items

Item and Readability Level (Flesch-Kincaid grade levels)Original Item and Source
Satisfaction (1=strongly disagree to 5=strongly agree)
In general, I am satisfied with the telemedicine system (11.5)Please rate your overall satisfaction with telemedicine (Dick)
My health is better than it was before I used the technology (5.8)My health is better than it was before I used the technology (ATA)
I am more involved in my care using the telemedicine system (8.0)I am more involved in my care using the telemedicine system (ATA)
The telemedicine system helps me to better manage my health and medical needs (9.4)Home technology helps me to better manage my health and medical needs (ATA)
The telemedicine system helps monitor my health condition (12.0)Using Tele Home Care, the nurse will be able to monitor my condition well (TMPQ)
My doctor uses information from the telemedicine system in my office visits (12.0)N/A
I follow my doctor's advice better since working with the telemedicine system (10.7)N/A
The telemedicine equipment is easy to use (10.7)The use of necessary equipment seems difficult to me (TMPQ)
I can always trust the equipment to work (3.7)I cannot always trust the equipment to work (TMPQ)
It was easy to learn to use the equipment (3.6)I was adequately trained to operate the equipment (ATA)
Talking to a nurse during a video visit is as satisfying as talking in person (9.1)I can be as satisfied talking to the nurse over the television as talking in person (TMPQ)
A nurse can get a good understanding of my medical problem during a video visit (9.9)A nurse can get a good understanding of my medical problem during a video visit (TMPQ)
My privacy is protected during video visits (12.0)Telecare can violate my privacy (TMPQ)
I can explain my medical problems well enough during a video visit (9.7)I am able to explain my medical problems well enough during a video visit (Demeris)
The lack of physical contact during a video visit is not a problem (8.5)I don't like that there is no physical contact during a Tele Home Care visit (TMPQ)
Video visits are a convenient form of healthcare delivery for me (8.7)Tele Home Care is a convenient form of health care delivery for me (TMPQ)
Video visits save me time (5.2)Tele Home Care saves me time (TMPQ)
Video visits make it easier for me to contact the nurse (5.8)Tele Home Care makes it easier for me to contact the nurse (TMPQ)
My nurse case manager answers my questions (5.6)My health care provider answers my questions (HCP)
My nurse case manager deals with my problems (3.7)My health care provider deals with my problems (HCP)
My nurse case manager engages me in my care (4.9)My health care provider engages me in my care (HCP)
Usefulness (1=not at all useful to 5=very useful)
ADA educational Web pages (12.0)N/A
Blood pressure testing (9.9)N/A
Blood sugar testing (9.9)N/A
Video visits from the nurse (4.4)N/A
Web site where I can review my results (2.2)N/A
  • Satisfaction: 1, strongly disagree to 5, strongly agree.

    TMPQ = Telemedicine Patient Questionnaire/Demeris1517; ATA = American Telemedicine Association Home Technology Survey Item Bank18; HCP = Engagement with Healthcare Provider20 Dick.19

Utilization. Home telemedicine unit use was tracked to capture number and length of video visits with nurse case managers, number of uploaded fingerstick glucose values and blood pressure values, number of visits to the data review Web site, and number of visits to the ADA educational Web site.

Data Collection Procedures

We recruited potential participants for the satisfaction and usefulness survey after receiving approval for an addendum to the initial IDEATel institutional review board approval. Other study measures were collected as part of the primary data collection. Participants were compensated for participation in the IDEATel RCT, but received no additional compensation for completion of the survey.

TSUQ. We used two methods of data collection for TSUQ. We mailed the English-version of the survey to all currently enrolled rural participants in the intervention arm of the IDEATel RCT. A member of the IDEATel staff called urban intervention arm participants and asked if they would be willing to complete the survey through an interview in the language of their choice in their home. We called those closest to the date of their annual IDEATel assessment (see below) and stopped after approximately half of those currently enrolled agreed to consider participating in the interview. Following the initial call, a bilingual physician (LG) telephoned potential participants, confirmed willingness to participate, and arranged for a time to visit the home to complete the survey. The survey questions were read to participants in either Spanish or English and they verbalized their answers or pointed to the answers on an enlarged version of the survey response scale.

IDEATel enrollment and annual assessments. For purposes of describing the study sample and examining the discriminant validity of the survey, we used selected sample characteristics (e.g., ethnicity, health beliefs, reasons for enrollment) collected during enrollment or annual assessments.

Utilization. Utilization was tabulated for each participant for the one year preceding the date of distribution of the perceived satisfaction and usefulness survey. We used these data to explore the relationships between utilization and perceptions of satisfaction and usefulness.

Data Analysis

Psychometric analyses. To examine the construct validity of the questionnaire, we conducted principal components factor analyses with Varimax rotation and Kaiser normalization. We first conducted a separate factor analysis for the urban and rural samples and then a combined sample factor analysis. We also examined construct validity using a discriminant validity approach in which we compared factor scores between groups hypothesized to differ (known group differences technique) using either independent samples t-tests or one-way analyses of variance as appropriate: ethnicity (Hispanic vs. non-Hispanic), health beliefs (diabetes not harmful vs. very harmful to health), reason for enrollment (believe technology can help with diabetes control—yes or no), and perceived quality of diabetes care during last 12 months. We calculated internal consistency reliability with Cronbach's alpha.

Descriptive, correlational analyses. We used descriptive, correlational analyses to characterize the study sample and examine relationships between utilization and perceptions of satisfaction and usefulness. We calculated mean scores for all survey items and created factor scores based on the results of the psychometric analyses.

Results

Sample

The response rate for the mailed survey was 66% and for the in-person interview, 87% of those called to ascertain interest in participating completed the survey. The sample was predominantly female (62.4%) and had a mean age of 72.8 years (Table 2). Most respondents in the rural sample were White, non-Hispanic (93.5%) and the majority of the urban respondents was Hispanic (81.4%) followed by 17.2% Black, non-Hispanic. The mean number of years with a diabetes diagnosis was 10.8. The majority of the sample did not take insulin (69.4%), but checked their glucose via fingerstick one (49.7%) or more (37%) times per day. More than half (59.2%) the respondents enrolled in IDEATel because they felt that telemedicine could help them control their diabetes. The urban as compared to the rural sample had significantly higher proportions of females, Hispanics, and participants indicating that they enrolled in IDEATel because they believed the technology could help. Urban participants also reported significantly lower ratings of quality of diabetes care in last 12 months. Rural participants had a significantly higher level of education and more frequently knew how to use a computer.

View this table:
Table 2

Characteristics of Study Sample and Comparisons between Rural and Urban Participants

VariableRural N=201 N (%)Urban N=145 N (%)Total N=346 N (%)
Gender *
   Male86 (42.8)44 (30.3)130 (37.6)
   Female115 (57.2)101 (69.7)216 (62.4)
Race/ethnicity **
   White, non-Hispanic188 (93.5)0 (0)188 (54.3)
   Black, non-Hispanic10 (5.0)25 (17.2)35 (10.1)
   Hispanic3 (1.5)118 (81.4)121 (34.0)
   Other0 (0)2 (1.4)2 (0.6)
Know how to use computer **
   Yes65 (33.3)6 (4.1)71 (20.1)
   No130 (66.7)139 (95.9)269 (76.2)
Take insulin
   Yes139 (69.7)101 (69.2)106 (30.6)
   No62 (30.8)44 (30.3)240 (69.4)
Frequency of glucose fingerstick
   Less than once per day25 (12.5)21 (14.5)46 (13.3)
   Once per day97 (48.3)75 (51.7)172 (49.7)
   More than once per day79 (39.3)49 (33.8)128 (37.0)
Reason for enrollment: telehealth can help me **
   Yes104 (52.0)101 (69.7)205 (59.2)
   No96 (48.0)44 (30.3)140 (40.5)
M (SD) M (SD) M (SD)
Age73.2 (6.8)72.1 (5.8)72.8 (6.4)
Number years education **7.0 (3.8)12.0 (3.3)9.9 (4.3)
Number years with diabetes diagnosis10.2 (9.4)11.1 (9.8)10.6 (9.6)
Self-rated sense of well being74.3 (15.0)72.1 (17.1)73.4 (15.9)
Quality of diabetes care last 12 months **3.3 (0.9)2.9 (0.9)3.1 (0.9)
  • Differences between rural and urban participants: * >.05, ** >.001.

Psychometric Analyses

Given the fact that the factor analyses for the mail and interview versions of the survey were similar, we report only the combined sample factor analysis. The principal components factor analysis extracted two components from the 21 satisfaction items explaining 63.6% of the variance (Table 3). The first factor, Video Visits, included 11 items and explained 34.3% of the variance. Ten items comprising the second factor, Use and Impact, explained 29.2% of the variance. In regards to discriminant validity, both factors, scores were significantly higher for Hispanics than for non-Hispanics. There were no significant differences related to health beliefs, reason for enrollment, or perceived quality of diabetes care during last 12 months. Cronbach's alpha was 0.96 for Video Visits and 0.92 for Impact and Use.

Perceptions of Satisfaction and Usefulness

All satisfaction items had a mean score of greater than 4 on a 5-point scale (Table 3). Those from urban areas reported significantly higher (p < 0.001) ratings of satisfaction (M = 52.6, SD = 4.0) with Video Visits than participants from rural areas (M = 46.9, SD = 6.4). Urban participants' Use and Impact ratings (M = 46.5, SD = 4.4) were also significantly higher (p < 0.001) than those of rural participants (M = 40.9, SD = 6.4).

View this table:
Table 3

Means, Factor Loadings and Explained Variance

ItemN M (SD)* Video VisitsUse and ImpactVariance Explained
A nurse can get a good understanding of my condition during a medical visit3384.5 (0.66)0.824
My nurse case manager answers my questions3404.6 (0.58)0.819
My nurse case manager deals with my problems3384.5 (0.66)0.805
My nurse case manager engages me in my care3384.5 (0.63)0.783
I can explain my medical problems well enough during a video visit3334.5 (0.66)0.742
The lack of physical contact during a video visit is not a problem3414.4 (0.69)0.722
My privacy is protected during video visits3104.5 (0.57)0.696
Talking to a nurse during a video visit is as satisfying as talking in person3404.5 (0.77)0.686
Video visits make it easier for me to contact the nurse3374.4 (0.78)0.674
Video visits are a convenient form of healthcare for me3424.5 (0.74)0.660
Video visits save me time3304.4 (0.82)0.631
34.1%
I am more involved in my care using the telemedicine system3414.4 (0.79) 0.773
The telemedicine equipment is easy to use3434.3 (0.85) 0.740
The telemedicine system helps me to better manage my health and medical needs3424.4 (0.74) 0.735
In general, I am satisfied with the telemedicine system3424.5 (0.71) 0.702
My health is better than it was before I used the technology3354.2 (0.93) 0.693
I follow my doctor's advice better since working with the telemedicine system3384.2 (0.90) 0.673
The telemedicine system helps monitor my health condition3424.5 (0.66) 0.642
It was easy to learn to use the equipment3404.2 (0.88) 0.603
My doctor uses information from the telemedicine system in my office visits2864.1 (1.1) 0.597
I can always trust the equipment to work3414.1 (1.1) 0.521
29.5%
Total Percent Variance Explained 63.6%
  • * Satisfaction: 1, strongly disagree to 5, strongly agree.

There were significant differences (p < 0.001) in Video Visit scores between those who knew how to use a computer at baseline (M = 46.8, SD = 5.7) and those who did not (M = 50.0, SD = 6.2). Use and Impact scores were also significantly higher (p < 0.009) for those who did not know how to use a computer at baseline (M = 44.1, SD = 6.5) than those who knew how to use a computer (M = 41.7, SD = 5.2). Educational level was negatively correlated with satisfaction with Video Visits (r = −0.356, p < 0.001) and Use and Impact (r = − 0.372, (p < 0.001).

In terms of perceived usefulness of the IDEATel components, urban participants rated all components higher than did rural participants. Overall, the feature rated as most useful was blood sugar testing (M = 4.9, n = 335), followed by blood pressure testing (M = 4.8, n = 334), video visits with nurse (M = 4.8, n = 332), results review Web site (M = 4.4, n = 266), and ADA educational Web pages (M = 4.1, n = 213). However, some rankings differed between the two groups. Both urban and rural participants ranked blood sugar testing as first, video visits with the nurse as third, and ADA educational Web pages as fifth. For urban participants, the Web site for reviewing results was ranked second and blood pressure testing tied for third in rank with video visits. In contrast, rural participants ranked the blood pressure testing component as second and the Web site for reviewing results as fourth.

Relationships between Utilization and Perceptions of Satisfaction and Usefulness

Mean frequency of utilization of IDEATel components was highest for blood sugar testing (M = 243.5, n = 337) followed by Web site for reviewing results (M = 106.6, n = 216), blood pressure testing (M = 94.4, n = 279), video visits (M = 10.7, n = 339), and ADA educational Web pages (M = 6.4, n = 61). Urban respondents uploaded fewer blood pressure (p = 0.001) and glucose (p < 0.001) values. They also used the Web site for reviewing results less frequently (p < 0.001) than rural respondents. There were no significant differences in number of visits to the educational Web site or in number of video visits. However, the mean time per visit (p < 0.001) and total video visit time (p < 0.001) were significantly less for urban than for rural participants.

Relationships varied between factor scores and the two IDEATel components (video visits and glucose value uploads) used by the most survey respondents. Ratings of usefulness of video visits with the nurse and number of video visits had a significant, positive correlation (r = 0.16, p = 0.004). Mean time per video visit had significant, but modest, negative correlations with both Video Visits (r = −0.17, p = 0.004) and Use and Impact (r = −0.24, p < 0.001). A one-way ANOVA showed no significant differences in factor scores among patients who tested and uploaded their glucose values less than 100 times, 100–400 times, or more than 400 times.

Discussion

TSUQ Development and Validation

The construct validity of the TSUQ is supported by a two-factor solution, Video Visits and Impact and Use, which explained 63.6% of the variance. One hypothesized difference in factor scores that provides evidence for the discriminant validity of the survey was the difference in factor scores of Hispanics as compared to non-Hispanics. Hypothesized differences related to health beliefs (enrolled because of belief that technology could help diabetes) and other satisfaction ratings (quality of diabetes care during last 12 months) were not supported. Internal consistency reliability of the two factors was excellent with Cronbach's alphas of 0.96 for Video Visits and 0.92 for Impact and Use.

As noted in recent reviews, most reports of telemedicine satisfaction do not include information on the reliability and validity of the questionnaire used to measure satisfaction.7,8,21 However, there are several notable exceptions. For example, Demeris and colleagues provided evidence for the internal consistency and test-retest reliability and validity (face, content, and construct) of the Telemedicine Perception Questionnaire (TMPQ).15 Yip et al. developed the Telemedicine Satisfaction Survey (TSQ), explored the construct validity through factor analysis, and documented the predictive validity of TSQ score for self-reported adherence, but not HbA1c.14 In a prison environment, factor analysis on a 14-item satisfaction questionnaire (n = 221) revealed two factors (information exchange and patient comfort) and adequate internal consistency reliabilities, 0.88 and 0.81 respectively.22 The findings of the psychometric analysis of TSUQ are equal or superior to those reported in these studies. Moreover, in contrast to existing telemedicine satisfaction surveys, TSUQ items evaluate a broad range of telemedicine services.

A number of reports have documented the extent of inadequate health literacy in the U.S. and its impact on health status and understanding of condition and treatment.2325 IDEATel educational materials were designed with careful attention to readability level. Consequently, our goal was to design TSUQ with a readability level no greater than 8th grade and this was achieved on average. However, there were several challenges. Items that included the word “telemedicine” in them had Flesch-Kincaid readability levels ranging from 8th to 12th grade. Given the familiarity of the participants with the term, the project team judged the readability of these items to be acceptable. Other telemedicine concepts also resulted in items with readability levels greater than 8th grade: blood pressure testing, blood sugar testing, ADA educational Web pages, and privacy. With the exception of the last, these were components of IDEATel that were familiar to participants. Given that privacy is a vital concept in telemedicine, further work may be needed to develop a TSUQ item with lower readability.

Perceptions of Satisfaction and Usefulness

Our study results indicate that, similar to other telemedicine studies, participants' perceptions of satisfaction and usefulness were highly positive.5,8 Urban participants had higher Video Visit and Use and Impact factor scores and also rated the usefulness of all IDEATel components more favorably than rural participants. There were no other studies located that specifically examined rural versus urban differences in telemedicine satisfaction and there is little convergence in the findings of studies that examined rural versus urban differences in perceptions of quality of care in general.26,27

Given that access to IDEATel was standard across urban and rural participants, a potential reason for these differences in perceptions of satisfaction and usefulness is survey administration method because data from urban participants was collected in a face-to-face interview rather than via a mailed survey as in the rural sample. The majority of the interviews were conducted by a physician from the Dominican Republic. Although she was not part of the IDEATel development team, the face-to-face contact or her position as a physician may have influenced some of the primarily Hispanic elders to respond more favorably. Another possible explanation is given that urban participants rated their diabetes care prior to IDEATel less positively than the rural participants, IDEATel may have been perceived as an improvement in diabetes care and consequently influenced perceptions of satisfaction and usefulness.

The role of ethnicity (Hispanic vs. non-Hispanic) on perceptions of satisfaction and usefulness is not conclusive because of the confounding with survey administration method. In the general patient satisfaction literature, several studies indicate that Hispanics have lower satisfaction levels than non-Hispanic whites.2831 Frequent sources of dissatisfaction include access or communication difficulties. These were not issues in IDEATel, because Spanish-speaking patients received video visits from Spanish-speaking nurse case managers and all other IDEATel materials were also available in Spanish. Similar to our results, Miceli found that although the underlying scale factor structure was similar, Spanish-language respondents as compared to English-language respondents reported higher mean satisfaction levels on a scale that was translated into Spanish.32

Relationships between Utilization and Perceptions of Satisfaction and Usefulness

In terms of the relationships between perceptions of usefulness and actual utilization of IDEATel components, although urban participants rated all components higher than rural participants, they uploaded significantly fewer blood pressure and glucose values and used the Web site for reviewing results less frequently than rural respondents. The interpretation of this relationship is confounded by the two different survey administration methods. The number of video visits was not significantly different between the two groups; this is not surprising since the number of visits was primarily driven by the study protocol for nurse case management. The negative relationship between mean video visit time and both Video Visit and Use and Impact factor scores was influenced by the fact that the higher mean visit times were experienced by rural participants who rated both factors lower than urban participants. The longer mean visit and total visit times were most likely a reflection of connectivity issues rather than true nurse case management time.

Strengths and Limitations

In contrast to the methodological concerns identified in many other studies, this study had an adequate sample size, included psychometric analyses that support the construct validity and internal consistency reliability of the TSUQ, included content related to patient-provider communication, had high response rates to the survey, and included evaluation of a range of telemedicine services. The primary limitation of the study is that the survey was administered differently in urban and rural participants, which prevented interpretation of differences between the two groups. We realized that the two methods of survey administration would confound interpretation of results between rural and urban samples, but the experience of the research team with the urban participants suggested that the response to a mailed survey would be poor. Given that comparison between urban and rural participants was not a purpose of the study, we chose a data collection method that was judged to likely result in an adequate sample to address the study questions.

Conclusion

Psychometric analyses support the construct validity and internal consistency reliability of TSUQ, which is available in both English and Spanish at a readability level of 8th grade. Both rural and urban participants reported high levels of satisfaction and found all IDEATel components useful. Further work is needed to examine the relationships between utilization and perceptions of satisfaction and usefulness and to explore the effect of location (urban vs. rural) and ethnicity on satisfaction with telemedicine services. In addition, it is essential that the readability of telemedicine satisfaction questionnaires be appropriately matched to the health literacy of the study sample to form the basis for valid and reliable measurements. Consequently, this aspect warrants further exploration and study.

Footnotes

  • The Informatics for Diabetes Education and Telemedicine (IDEATel) project is supported by Cooperative Agreement 95-C-90998 from the Centers for Medicare and Medicaid Services. The development of the satisfaction survey and data collection and analysis of the satisfaction data were supported by the Center for Evidence-based Practice in the Underserved (P20 NR007799).

References

View Abstract