An Audit of the Appropriateness of Teletriage Nursing Advice

8 pages

English

An Audit of the Appropriateness of Teletriage Nursing Advice

Thowyug

Le téléchargement nécessite un accès à la bibliothèque YouScribe
Tout savoir sur nos offres

8 pages

English

Le téléchargement nécessite un accès à la bibliothèque YouScribe
Tout savoir sur nos offres

A propos
Informations
Extrait

Description

TELEMEDICINE JOURNAL AND e-HEALTHVolume 10, Number 1, 2004© Mary Ann Liebert, Inc.An Audit of the Appropriateness of Teletriage Nursing Advice1 2JOHN C. HOGENBIRK, M.Sc., and RAYMOND W. PONG, Ph.D.ABSTRACTThis study assessed the appropriateness of advice given by teletriage nurses to patients innorthern Ontario. Assessments used audiotapes and printed records of 73 calls, selected fromapproximately 350 calls based on sound quality, completeness, and consent of caller and tele-triage nurse. Audits were conducted independently by one family physician, one nurse prac-titioner, and one registered nurse with teletriage experience. In 56% of the 73 calls, all threeauditors judged the nurse’s advice as “appropriate.” In 92% of the 73 calls, at least two of thethree auditors judged the teletriage nurse’s advice as “appropriate.” All calls were rated as“appropriate” by at least one auditor. If not “appropriate,” then auditors were three timesmore likely to rate the advice as “overly-cautious” rather than “insufficient.” The percentageof calls with the same rating varied from 62% to 86% with an outlier of 33%. Nurse practi-tioners tended to rate the appropriateness of the advice slightly, but significantly lower thanthe rating given by family physicians or registered nurses. Interestingly, nurse practitionerstended to rate aspects of the nurse–caller interaction advice as slightly and significantly bet-ter than the rating chosen by family physicians or ...

Informations

Publié par	Thowyug
Nombre de lectures	81
Langue	English

Extrait

TELEMEDICINE JOURNAL ANDe-HEALTH Volume 10, Number 1, 2004 © Mary Ann Liebert, Inc.

An Audit of the Appropriateness of Teletriage Nursing Advice

1 2 JOHN C. HOGENBIRK, M.Sc., and RAYMOND W. PONG, Ph.D.

ABSTRACT

This study assessed the appropriateness of advice given by teletriage nurses to patients in northern Ontario. Assessments used audiotapes and printed records of 73 calls, selected from approximately 350 calls based on sound quality, completeness, and consent of caller and tele-triage nurse. Audits were conducted independently by one family physician, one nurse prac-titioner, and one registered nurse with teletriage experience. In 56% of the 73 calls, all three auditors judged the nurses advice as “appropriate.” In 92% of the 73 calls, at least two of the three auditors judged the teletriage nurses advice as “appropriate.” All calls were rated as “appropriate” by at least one auditor. If not “appropriate,” then auditors were three times more likely to rate the advice as “overly-cautious” rather than “insufficient.” The percentage of calls with the same rating varied from 62% to 86% with an outlier of 33%. Nurse practi-tioners tended to rate the appropriateness of the advice slightly, but significantly lower than the rating given by family physicians or registered nurses. Interestingly, nurse practitioners tended to rate aspects of the nurse–caller interaction advice as slightly and significantly bet-ter than the rating chosen by family physicians or registered nurses. The teletriage service was providing appropriate advice, but the generalizability of these results may be limited be-cause of the selection of calls.

INTRODUCTION

ELEPHONE HEALTH SERVICEShave been used T as an integral component of the primary care system in many parts of the world. They are often intended to enhance access to care, to help reduce unnecessary use of more costly ser-vices, and to encourage self-care and informal care, especially where health care resources are scarce and access to care is inadequate. There are three broad and overlapping types of telephone service: (1) health hotlines that provide information only; (2) teletriage services that provide health advice, self-care instruc-

tions, and recommendations as to which types of health care should be accessed; and (3) tele-phone management services that act as formal gatekeepers to the health care system. Regard-less of their nature, assessing the appropriate-ness of the information provided is necessary to ensure that telephone health services meet acceptable standards. This study assessed the appropriateness of advice given by teletriage nurses to patients in northern Ontario. A teletriage pilot project, called Direct Health/ TéléSanté, ran for 22 months, from June 1, 1999, to March 31, 2001, in northern Ontario, north of, and including, the Districts of Nipissing and

1 2 Senior Researcher and Research Director, Centre for Rural and Northern Health Research, Laurentian University, Sudbury, Ontario, Canada.

Parry Sound. It is a geographically vast area 2 (800,000 km ) with slightly over 800,000 in-habitants. The teletriage services were pilot-tested in northern Ontario prior to becoming a province-wide program. As part of the pilot testing, a comprehensive evaluation was con-ducted by researchers at the Centre for Rural and Northern Health Research (CRaNHR), Laurentian University. The audit of the appro-priateness of advice provided by the teletriage nurses, as reported in the present study, was one component of a broader evaluation. The pilot project was conducted by Clinidata Corporation out of a call center in North Bay, Ontario. Offered in English and French and available 24 hours a day, 7 days a week, the teletriage services were accessible through a toll-free telephone line for residents of north-ern Ontario. The service advised people with nonurgent symptoms who were concerned about a health problem and/or uncertain about what to do about it. Callers spoke with a reg-istered nurse who used computer-assisted and medically approved guidelines and nursing ex-perience to query patients on their symptoms and recommend the most appropriate type of care. Depending on the nature and severity of the patients symptoms, the nurse provided recommendations for self-care or informal care, visit a family doctor or walk-in clinic, or visit the nearest emergency department (ED). Nurses also provided basic health information and in-formation about prescriptions and over-the-counter medications. There are several approaches to the evalua-tion of a teletriage service and the choice of out-comes. Ultimately, one would be interested in whether the service influences the long-term health status or well-being of patients who use it. A systematic review of the literature identi-fied 10 studies that typically reported no dif-ferences in deaths, hospital admissions, or number of visits to EDs as compared to other 1 service options. Attribution is an issue here, since several other factors may influence mor-tality and morbidity. Typically, an evaluation of a teletriage ser-vice is focused on changes in health status within a few days of the call or differences in nursing assessment or medical diagnosis based 2,3 on a review of call documentation, audio

HOGENBIRK AND PONG

4 tapes, or results of a face-to-face encounter 5 with the patient. Alternatively, the evaluation may examine health services use within a few days of the call and whether the advice was consistent with use, with possible adjustments for changes in health status that may have oc-6–9 curred after the call. Other evaluation meth-ods have compared the level of agreement be-tween teletriage nurses and other health care practitioners who have seen the same patients 10–12 face-to-face. Standardized patients have been used to assess agreement using teletriage 13,14 nurses or agreement between nurses and other practitioners in providing teletriage ser-4 vices or between teletriage nurses and ac-cepted recommendations for selected case his-13,15 tories. Three main elements were considered in this assessment of teletriage nursing advice: (1) the patients (or those calling on behalf of the pa-tients); (2) the teletriage nurse; and (3) the com-puter-assisted clinical guidelines. The patients willingness and ability to comply with the nurses advice can affect the accuracy of the nursing assessment and influence the nature of the advice. Similarly, the nurses ability to ex-tract pertinent information on the patients con-dition and circumstances will affect the nature of the advice. It is also important to assess the usefulness of the clinical software in facilitat-ing the communication between the teletriage nurse and the patient, and in assisting the nurses clinical decision-making.

MATERIALS AND METHODS

The appropriateness of the advice was assessed for 73 tape-recorded calls, supple-mented by a print-out of the computerized call record, which included data on: (1) patient de-mographics, (2) clinical information, and (3) recommendation. The clinical information sec-tion contained data on the callers relationship to patient (i.e., was the caller calling for herself or for someone else?), a brief health history, presenting problem, a set of nursing assess-ment questions, the care advice given to the pa-tient, and a closing statement. The closing state-ment advised patients on what to do or how to monitor the situation and how to get help if

APPROPRIATENESS OF TELE-NURSING ADVICE

things got worse. The recommendation section included the name of clinical guideline used and the recommendation provided by the tele-triage nurse. There were about 350 calls tape-recorded by Clinidata at the North Bay site from May 4 to June 7, 2001. Approximately 300 calls were recorded from the start to the finish of the con-versation. The teletriage nurse asked for the callers consent in 112 calls (37%) and the caller granted consent in 101 calls (90% consent rate for those callers who were asked). These 101 calls were handled by 23 teletriage nurses, and 15 (65%) of these nurses granted consent. At the end of the selection process, there were 73 tape-recorded calls handled by 14 teletriage nurses. There was a range of 1–15 calls per nurse with a mean of approximately 5 calls per nurse. Identifying information was deleted to protect privacy. Patients age and community of residence, important in the assessment of the appropriateness of advice, were made available to the auditors. The study was approved by Laurentian Universitys Research Ethics Board. There were six auditors: two family physi-cians (FPs), two nurse practitioners (NPs), and two registered nurses (RNs) with teletriage ex-perience, chosen because the teletriage service focused on everyday health problems and en-hancing access to primary care. The RN audi-tors had previous experience in providing tele-triage services. All auditors were practicing and licenced by their respective licencing bod-ies during the audit period. Audits were conducted independently, and each call was assessed by one FP, one NP, and one RN. Tape-recorded calls were systemati-cally assigned such that each auditor would as-sess one-half of all calls in the sample. How-ever, seven calls were reassigned from one NP to the other NP due to time constraints. Tele-triage nurses with more than three tape-recorded calls were evaluated by both FPs, both NPs, and both RNs. For example, a teletriage nurse with seven tape-recorded calls had three calls audited by one FP and four calls audited by the other FP. Similar arrangements were made for assigning calls to NPs and RNs. Auditors were unable to assess the entire computerized decision support software pack-age (Sharp Focus, HealthLineSystems Inc. San

Diego, CA) because of copyright restrictions. Auditors, however, assessed certain compo-nents of the software that applied directly to the tape-recorded calls. The audit form con-tained 16 main questions, but the focus in this paper is on the auditors assessment of the ap-propriateness of the advice given by the nurse to the patient as well as evaluating aspects of the nurse-patient interaction. The appropriate-ness of the advice was assessed on a seven-point scale. Aspects of the nurse-patient inter-action were rated on a five-point scale. The auditors responses to open-ended questions were clarified by the research team after dis-cussion with the auditors. The audit process and audit form were pretested by two FPs and two NPs. The analysis was descriptive in na-ture, using Pearsons Chi-squared statistic for contingency tables,t-tests, and percent agree-ment for estimates of interauditor agreement.

RESULTS

Description of calls, callers, and patients The teletriage nurses recommended self-care (including informal care to others) in 34% of the 73 calls, and advised 29% of the patients to see a physician within 4 hours (urgent physi-cian referral category) (Table 1). A comparison to recommendations for all calls logged during

TABLE1. RECOMMENDATIO NSUGGESTED BYCLINICALGUIDELINE

Recommendation

Priority (911) Emergent (ED) Urgent physician referral Physician referral Interim care (self-care) Information only Other recommendation No recommendation Total

Audited calls n5(%)

1 (1%) 21 (29%) 14 (19%)

3 (4%) 25 (34%) 8 (11%) 0 (0%) 1 (1%) c 73 (100%)

a All calls n5(%)

2,307 (2%) 22,811 (23%) 26,066 (26%)

8,018 (8%) 38,727 (28%) b 3,288 (3%) 0 (0%) 101,217 (100%)

a 16 Clinidata Corporation (2001). b Included in self-care. c 2 x 55.017,df54,p50.276, for 9111ED, Urgent physician referral, Physician referral, interim care1in-formation only, and other1no recommendation.

the pilot project by the service provider (Clin-idata Corporation) did not find a statistically 2 significant difference (x55.107,df54,p5 0.276). The teletriage nurses accessed 41 guide-lines. Approximately 63% of the guidelines were accessed once, 15% were accessed twice, and 22% were accessed three times. There were 15 pediatric guidelines used in 30 calls and these were the most frequently used guide-lines. Guidelines for trauma and genitourinary symptoms were also frequently accessed. Dif-ferent health information topics were accessed in 14 calls. Guidelines and health topics used in the audited calls were representative of those used during the pilot project. Most callers were female (86%) and most (60%) had called for another person, typically a young child. The proportion of female callers in the audit sample was not statistically signif-icantly different from that obtained from a sur-vey of callers (2389 questionnaires returned, 5475 mailed) conducted by the research team 2 as part of the evaluation (x50.001,df51,p5 0.980). Most patients were female (67%), and the proportion was not statistically signifi-cantly different from that obtained by the sur-2 vey (x50.875,df51,p50.350) or from a 2 larger sample of 27,302 call records (x50.443, df51,p50.505). There were, however, signif-icantly (p,0.001) more female callers or fe-male patients in the audit sample as compared to the population of northern Ontario, where ap-proximately 50% of the population are female. The average age of patients was 19 years. The majority were 17 years of age or younger (56%), while 21% were aged 17–34 years (Table 2). The

Age of patient (years)

Under 17 17–34 35–49 50 or older Subtotal Missing Total

TABLE2.

Audit patients

b 41 (56%) 15 (21%) 10 (14%) 7 (10%) 73 0 73

HOGENBIRK AND PONG

age distribution of audited patients was not sta-tistically significantly different from that ob-2 tained by the survey (x52.748,df53,p5 2 0.432) or from the call records (x53.373,df5 3,p50.338). There were, however, significantly more audited patients in the 17 or younger age group as compared to the population of north-2 ern Ontario (x544.128,df53,p,0.001).

Appropriateness of advice In 56% of the 73 calls, all three auditors judged the nurses advice as “appropriate.” In 92% of the 73 calls, at least two of the three au-ditors judged the teletriage nurses advice as “appropriate.” All calls were rated as “appro-priate” by at least one auditor. If the auditors did not rate the teletriage nurses advice as “ap-propriate,” then they were approximately three times more likely to rate it as “overly-cautious” (23 calls) rather than “insufficient” or “inap-propriate” (8 calls). The overall mean rating of the 73 calls was 4.1 (15“inappropriate,” 45“appropriate,” 75“unnecessarily overly cautious”) (Table 3). Mean rating given by NPs (3.8) was statistically significantly different (p,0.001) from that given by the RNs (4.2) or FPs (4.3) (pairedt-tests). Mean ratings given by RNs and FPs were not significantly different (p50.109) from one another. Interrater agreement, calculated as the per-centage of calls rated exactly the same by each pair of auditors, ranged from 33% to 86% (Table 4). Including audits in which one auditor was within one rating category of the other auditor

AGE OFPATIENT BYDATASOURCE

Survey patients

1162 (49%) 477 (20%) 353 (15%) 384 (16%) c 2376 13 2389

Call records

12,975 (46%) 8091 (29%) 3825 (14%) 3041 (11%) d 27,932 104 28,036

a Source: Statistics Canada 1996 Census. b Percentages were calculated separately for each column based on subtotals. c 2 x 52.748,df53,p50.432 (comparison of audit and survey). d 2 x 53.373,df53,p50.338 (comparison of audit and call records). e 2 x 544.128,df53,p,0.001 (comparison of audit and 1996 population).

a 1996 Population

197,005 (24%) 207,605 (25%) 193,180 (23%) 228,485 (27%) e 826,275 0 826,275

a AUDITORS RATING OF THETELETRIAGENURSES ADVICE TOPATIENTS

APPROPRIATENESS OF TELE-NURSING ADVICE

13 of 18 (72%) 6 of 18 (33%) 11 of 14 (79%) 15 of 23 (65%)

a Self-agreement statistics were not presented. b Agreement statistics could not be calculated for (1) FP1 and FP2, (2) NP1 and NP2, or (3) RN1 and RN2 because there was no overlap.

67 (92%)

13 of 17 (76%) 15 of 18 (83%) 9 of 14 (64%) 18 of 21 (86%)

FP n(%)

TABLE3.

Appropriateness of the teletriage nurses advice to the patient

10 of 13 (77%) 11 of 16 (69%) a

NP1

NP2

FP1

FP2

Auditors

Auditor

There were 22 calls for which one of the au-ditors suggested a level of care that differed from that of the teletriage nurse. These include the 18 calls for which advice was not appro-priate or did not have mitigating circum-stances, and four calls with mitigating circum-stances for which the auditor still suggested an alternative level of care. In 73% of these 22 calls, the auditor suggested a level of care that was less urgent than what the teletriage nurse had advised. For example, the auditor recom-mended that the patient should see their physi-cian within 24 hours whereas the teletriage nurse had recommended a visit to the ED or to the physician within 4 hours (six calls). In an-other four calls, the auditor suggested that self-care would suffice, whereas the teletriage nurse had suggested a visit to a physician within 24 hours. In the remaining 27% of the 22 calls, the au-ditor suggested a level of care that was more urgent. Four calls for which the teletriage nurse advised self-care or provided information were upgraded to a physician visit. The advice for another four calls was upgraded to an ED visit. In 82% of these 22 calls, it was only one audi-tor who disagreed with the level of urgency suggested by the teletriage nurse. Mitigating circumstances were cited for 21 calls. These include the 18 calls that had “mit-igating circumstances,” and three additional

Total n(%)

3 (1%) 4 (2%) 2 (1%) 181 (83%) 16 (7%) 8 (4%) 3 (1%) 217 4.1 (0.469)

3 (4%) 3 (4%)

NP n(%)

RN2

NP2

16 of 22 (73%) 13 of 21 (62%) b

RN1

TABLE4. EXACTAGREEMENTBETWEEN PAIRS OFAUDITORS

NP1

yielded a modified overall agreement of 76–94%. An examination of how auditors rated the tele-triage nurses advice (Table 3) and percent agreement between auditors (Table 4) suggests that ratings were reasonably robust. Paradoxi-cally, the high level of agreement in one cate-gory (as opposed to high level of agreement in several categories) resulted in low values for correlation coefficients (Pearsons or Spear-17 mans) as well as low values for Cohens kappa. Cohens kappa (a measure of interrater agree-ment) ranged from20.19 to 0.23 (data not pre-sented), which is considered to be poor to slight 18 agreement.

RN n(%)

c 1 (1%) 55 (76%) 8 (11%) 6 (8%) 2 (3%) d 72 4.3 (0.772)

73 3.8 (0.706)

1 (1%) 1 (1%) 59 (82%) 8 (11%) 2 (3%) 1 (1%) d 72 4.2 (0.628)

b Inappropriate (1) Insufficient, no mitigating circumstances Insufficient but with mitigating circumstances Appropriate Overly cautious but with mitigating circumstances Overly cautious, no mitigating circumstances Unnecessarily overly cautious Total Mean rating (7 point scale) (SD)

a Zeroes were removed from the table to aid interpretation. b Value on 7-point scale. c Percent of audits were calculated down columns. d One call was not rated.

Rating Scale

5 (2%) 15 (7%)

Caller became less anxious

15 (7%) 19 (9%)

Caller became slightly more anxious

70 (33%) 73 (33%)

No change in callers anxiety level

3.6 (0.590) 3.5 (0.602)

79 (36%) 71 (33%)

Caller became slightly less anxious

98 (46%) 105 (48%)

3.7 (0.470) 3.6 (0.537)

Mean rating (5-point scale) (SD)

214

215

218

Total n

73 (34%)

90 (42%)

31 (15%)

218

Total

212

27 (12%) 23 (11%)

29 (13%) 16 (7%)

17 (8%)

Good to excellent n(%)

Poor to good n(%)

Good n(%)

3.6 (0.628)

Mean rating (5-point scale) (SD)

1 (1%)

calls in which the auditor judged the advice as “appropriate.” The most frequently cited miti-gating circumstance was that the patient did not have a physician (seven calls) followed by the absence of an after-hours or walk-in clinic (AHC) (six calls) in the patients community. Overall, access-related mitigating circumstances (e.g., no physician, ED, or AHC) were cited in 13 of the 21 calls while patient-related mitigat-ing circumstances (e.g., difficulty understand-ing, unwilling or unable to comply) were cited in eight calls. Two calls had both access and pa-tient-related mitigating circumstances.

standing was 3.5–3.7 (15“poor,” 55“excel-lent”) (Table 5). Most ratings occurred in the categories of “good to excellent” (46–48%) and “good” (33–36%). Mean rating of the change in the callers anxiety level was 3.6 (15“more anxious,” 35“no change,” 55“less anxious”). Most of the ratings occurred in the category of “slightly less anxious” (42%) or “no change” (34%). NPs gave the highest mean rating (3.8–4.1) in all five aspects of the nurse–caller interaction, whereas RNs typically gave the lowest mean rating (3.0–3.6). There were sta-tistically significant differences (p,0.05) for all pair-wise comparisons of auditors (e.g., NP vs. FP, NP vs. RN, FP vs. RN) with the excep-tion of the change in anxiety, where NPs (mean53.8) gave significantly higher ratings than FPs (3.5) but not RNs (3.6).

Each of the 73 calls was audited by one FP, one NP, and one RN. In 92% of the 73 calls, at

The auditors rated five aspects of the tele-triage nurse–caller interaction. Mean rating of the: (1) teletriage nurses ability to establish a relationship with the caller; (2) nurses ability to extract pertinent information; (3) callers ability or willingness to describe the problem, and (4) the callers level of literacy or under-

b 3 c (1%) 3 (1%)

a Poor (1) n(%)

3 (1%) 5 (2%)

Teletriage nursess abililty to Establish a relationship with the caller/patient Extract pertinent information Callers Abililty or willingness to describe the problem Literacy level or level of understanding

AUDITORSOVERALLIMPRESSION OF THETELETRIAGENURSE-CALLERINTERACTION

Evaluation of the teletriage nurse–caller interaction

DISCUSSION

Caller became more anxious a (1)

HOGENBIRK AND PONG

TABLE5.

Callers anxiety Overall change in callers level of anxiety

104 (48%) 102 (48%)

a Value on 5-point scale. b Number of audits. c Percent of audits calculated across rows.

Excellent n(%)

APPROPRIATENESS OF TELE-NURSING ADVICE

least two of three auditors rated the teletriage nurses advice as “appropriate.” This is com-parable with an evaluation of teletriage refer-rals to a childrens hospital ED in Alabama, where Barberet al. reported that two of three physician auditors had agreed that 80% of the 9 recommended visits were appropriate. The variation among auditors in the assessment of the appropriateness of the teletriage nurses ad-vice may be due to the different perspective that the family physicians, nurse practitioners, and registered nurses brought to the audit pro-cess. A certain amount of variability is also to be expected among teletriage nurses. For ex-ample, Belman and colleagues found a mean agreement of 83% (range 64–100%) among 15 teletriage nurses in Colorado based on 15 dif-ferent case scenarios from standardized (mock) 13 patients. In our study, the percentage of calls that were rated exactly the same by each pair of auditors varied from 62–86% with an outlier of 33%. In the calls in which at least one auditor did not rate the teletriage nurses advice as “ap-propriate,” the auditor was three times more likely to rate it as “overly cautious” rather than “insufficient” or “inappropriate.” These results were consistent with Clinidatas policy and with the Canadian Nurses Associations rec-19–21 ommendations to err on the side of caution. Access-related mitigating circumstances (e.g., no FP or ED or AHC) were most frequently cited followed by patient-related circumstances (e.g., difficulty understanding, unwilling or un-able to comply). Patient understanding and willingness or ability to comply are crucial to 22–24 the process. In over 80% of the audits, the auditors rated the teletriage nurses ability to establish a relationship with the caller and ex-tract pertinent information as “good to excel-lent” or “good.” The callers ability or willing-ness to describe the problem and the callers level of literacy or understanding was also rated as “good to excellent” or as “good” in ap-proximately 80% of the audits. The auditors also judged that the caller became slightly less anxious in 42% of the audits. No change in the callers level of anxiety was recorded for 34% of the audits. The major limitation to the generalizability of results was the fact that the audited calls

were not randomly selected due to practical limitations and informed consent require-ments. However, the characteristics of the calls, callers, and patients were comparable to those of the pilot project in terms of call disposition, guidelines, and information topics used, as well as callers or patients demographic char-acteristics such as age and gender. However, more callers and patients are female and are younger than the population of northern On-tario. This study did not specifically address is-sues of accuracy of the nursing assessment as determined by subsequent changes in patients health status or use of health services. Nor did the study examine all aspects of safety and quality, unless directly related to appropriate-ness of advice. However, Clinidata Corpora-tion has a training program and a quality im-provement program that seeks to address these 25 concerns. Overall, the teletriage nurses advice was rated as “appropriate” by at least two of three auditors in over 90% of the audited calls. In ap-proximately three-quarters of the 31 calls in which the teletriage nurse and auditor dis-agreed on the level of urgency, the teletriage nurse erred on the side of caution by advising a level of care that was, in the auditors opin-ion, overly cautious. The “overly cautious” na-ture of the advice given should be understood in the broader context of the Canadian health care system. Unlike some teletriage programs in managed care settings in the United States or in the National Health System of the United Kingdom, teletriage programs in Canada are not gatekeepers. Thus, there was no pressure on teletriage nurses to direct patients away from more costly services such as physician or ED visits. As a matter of fact, in Ontario and some other provinces, teletriage programs, physician services, and emergency services are run and funded separately. Over utilization of, for instance, ED services has no adverse finan-cial implications for a teletriage program. On the other hand, if a patients situation deterio-rates after he has been told by a teletriage nurse that there is no need to see a doctor, the tele-triage program could be blamed for the conse-quence. Thus, there appears to be some incen-tive for teletriage nurses to err on the side of caution when in doubt.

ACKNOWLEDGMENTS

We would like to thank The Richard Ivey Foundation for funding the research program and thank the auditors for their thorough par-ticipation. We thank the nurses and other per-sonnel at Clinidata Corporation for their sup-port and advice and thank our colleagues at CRaNHR for their assistance.

REFERENCES

1. Stacey D, Noorani HZ, Fisher A, Robinson D, Joyce J & Pong RW.Telephone triage services: systematic review and survey of Canadian call centre programs. Ottawa, On-tario: Canadian Coordinating Office for Health Tech-nology Assessment. Technical report no. 43. Novem-ber,2003. 2. Gaffney P, Crane S, Johnson G, Playforth M. An anal-ysis of calls referred to the emergency 999 service by NHS Direct.Emerg Med J2001;18:302–304. 3. Poole SR, Schmitt BD, Carruth T, Peterson-Smith A, Slusarski M. After-hours telephone coverage: the ap-plication of an area-wide telephone triage and advice system for pediatric practices.Pediatrics1993;92:670– 679. 4. Perrin EC and Goodman HC. Telephone management of acute pediatric illnesses.NEJM1978;298:130–135. 5. Allen-Davies JT, Beck A, Parker R, Ellis JL, Polley P. Assessment of vulvovaginal complaints: Accuracy of telephone triage and in-office diagnosis.Obstet Gy-necol2002;99:18–22. 6. Marsden J. An evaluation of the safety and effective-ness of telephone triage as a method of patient prior-itization in an ophthalmic accident and emergency service.J Adv Nurs2000;31:401–409. 7. Lee TJ, Baraf LJ, Guzy JJD, Woo, H. Does telephone triage delay significant medical treatment.Arch Pedi-atr Adolesc Med2003;157:635–641. 8. Frisbee SJ, Malloy M, Meurer JR, Kuhagen KA & Kini NM. Urban Wisconsin pediatric patients using an af-ter-hours telephone triage service: Outcomes and compliance.WMJ2001;100:55–58. 9. Barber JW, King WD, Monroe KW, Nichols MH. Eval-uation of emergency department referrals by tele-phone triage.Pediatrics2000;105:819–821. 10. Kempe A, Dempsey C, Hegarty T, Frei N, Chandra-mouli V, Poole SR. Reducing after-hours referrals by an after-hours call center with second-level physician triage.Pediatrics2000;106:226–230. 11. Kempe A, Dempsey C, Whitefield J, Bothner J, MacKenzie T, Poole S. Appropriateness of Urgent Re-ferrals by Nurses at a Hospital-Based Pediatric Call Center.Arch Pediatr Adolesc Med2000;154:355–360.

HOGENBIRK AND PONG

12. Kempe A, Luberti A, Belman S, Hertz A, Sherman H, Amin D, Dempsey C, Chandramouli U, MacKenzie T. Outcomes associated with pediatric after-hours care by call centers: a multicenter study.Ambulat Pe-diatr2003;3:211–217. 13. Belman S, Murphy J, Steiner JF, Kempe A. Consis-tency of triage decisions by call center nurses.Ambu-lat Pediatr2002;2:396–400. 14. Wachter DA, Brillman JC, Lewis J, Sapien RE. Pedi-atric telephone triage protocols: standardized deci-sionmaking or a false sense of security?Ann Emerg Med1999;33:388–394. 15. Rupp RE, Ramsey KP, Foley JD. Telephone triage: re-sults of adolescent clinic responses to a mock patient with pelvic pain.J Adolesc Health1994;15:249–253. 16. Clinidata Corporation. Direct Health Report for the City of North Bay. June 1999–March 2001 Report. Clinidata Corporation,2001. 17. Uebersax, JS. Kappa coefficients: A critical appraisal. InStatistical methods for rater agreement. Downloaded July 18, 2002 fromhttp://ourworld.compuserve. com/homepages/jsuebersax/agree.htm. 18. Landis JR, Koch GG. The measurement of observer agreement for categorical data.Biometrics1977;33: 159–174. 19. Clinidata Corporation. Guidelines and Procedures Manual. #3.1.1.5 Scope of Practice v2. Clinidata Cor-poration. June,2001. 20. Clinidata Corporation. Guidelines and Procedures Manual. #3.3.2 Transfer to Call to 911 Ambulance Dis-patch—Ontario. Clinidata Corporation. June,2001. 21. Canadian Nurses Association. Telehealth: Great po-tential or risky terrain?Nursing Now2000;9:5–8. 22. Leclerc BS, Dunnigan L, Cote H, Zunzunegui MV, Ha-gan L, Morin D. Callers ability to understand advice received from a telephone health-line service: com-parison of self-reported and registered data.Health Serv Res2003;38:697–710. 23. Cooke MW, Wilson S, Cox P, Roalfe A. Public un-derstanding of medical terminology: non-English speakers may not receive optimal care.J Accident Emerg Med2000;17:119–121. 24. Larson-Dahn ML. Tel-eNurse practice—quality of care and patient outcomes.JONA2001;31:145–152. 25. Clinidata Corporation. Guidelines and Procedures Manual. Clinidata Corporation,2001. Address reprint requests to: John C. Hogenbirk Centre for Rural and Northern Health Research Laurentian University 935 Ramsey Lake Road Sudbury, Ontario P3E 2C6, Canada

E-mail:jhogenbirk@laurentian.ca