6 pages

English

Guide to benchmark reports

Effo - R. Jules

Le téléchargement nécessite un accès à la bibliothèque YouScribe
Tout savoir sur nos offres

6 pages

English

Le téléchargement nécessite un accès à la bibliothèque YouScribe
Tout savoir sur nos offres

A propos
Informations
Extrait

Description

Guide to benchmark reports 2008 Category C Ambulance service user survey Benchmark reports are produced for most surveys to show how the survey results for each trust participating in a particular survey compares with the results from all other trusts. This guide is divided into five sections: Section one: describes the benchmark reports Section two: describes how to use the benchmark reports and the limitations of the data Section three: describes how to understand the data Section four: provides guidance on using the benchmark reports to make comparisons between trusts Section five: describes how the data in the benchmark reports is calculated 1). Description of the reports The graphs included in the reports display the scores for a trust, compared with national benchmarks. Each bar represents the range of results for each question across all trusts that took part in the survey. The score for a trust is represented by a white diamond. The line on either side of the diamond shows the amount of uncertainty surrounding the trust’s score, as a result of random fluctuation. These are known as lower and upper confidence intervals. Please see section three below for more detailed information about confidence intervals. An example of a set of graphs from a benchmark report for trust X can be seen in chart one below. For the first question (Was the ambulance control room operator reassuring?) it can be seen that nationally, scores varied ...

Informations

Publié par	Effo
Nombre de lectures	51
Langue	English

Extrait

Guide to benchmark reports

2008 Category C Ambulance service user survey

Benchmark reports are produced for most surveys to show how the survey results for each trust

participating in a particular survey compares with the results from all other trusts.

This guide is divided into five sections:

Section one:

describes the benchmark reports

Section two:

describes how to use the benchmark reports and the limitations of the data

Section three

: describes how to understand the data

Section four:

provides guidance on using the benchmark reports to make comparisons between

trusts

Section five:

describes how the data in the benchmark reports is calculated

1). Description of the reports

The graphs included in the reports display the scores for a trust, compared with national

benchmarks. Each bar represents the range of results for each question across all trusts that took

part in the survey.

The score for a trust is represented by a white diamond. The line on either side of the diamond

shows the amount of uncertainty surrounding the trust’s score, as a result of random fluctuation.

These are known as lower and upper confidence intervals. Please see section three below for

more detailed information about confidence intervals.

An example of a set of graphs from a benchmark report for trust X can be seen in chart one

below. For the first question (Was the ambulance control room operator reassuring?) it can be

seen that nationally, scores varied between 87 (the lowest score) and 99 (the highest score).

Trust X scored 95 for this particular question. The lower confidence interval is 91 and the upper

confidence interval is 99.

Chart 1

2). How the benchmark reports should be used

Benchmark reports should be used to identify how a trust is performing in relation to all other

trusts that took part in the survey. From this, areas for improvement can be identified.

Limitations of the data

Because the average scores for each trust are estimates based on a sample of patients rather

than

all

patients at the trust, it is very often impossible to separate the performance of trusts.

That

is, in many cases the differences between trusts’ mean scores will not be

statistically significant

This means that if we were to repeat the survey with a different sample of patients, we would not

be confident that the results would show the same differences.

As such, data used in the

benchmark reports is fundamentally

not

suitable for generating league tables.

Also, it should be noted that the data only show performance relative to other trusts: there are no

absolute thresholds for ‘good’ or ‘bad’ performance.

Thus, a trust may score lowly relative to

others on a certain question whilst still performing very well on the whole.

This is particularly true

on questions where the majority of trusts score very highly.

3.) Understanding the data

Since the score is based on a sample of patients in a trust rather than

all

patients, the score may

not be exactly the same as if everyone had been surveyed and had responded. To account for

this, a ‘confidence interval’ is calculated.

For each trust, then, the benchmark report shows three

values for each question – an average (‘mean’) score as well as its lower (‘lcl’) and upper (‘ucl’)

confidence limits.

A confidence interval is calculated as an indication of the range within which the ‘true’ score would

lie if all patients had been surveyed. The confidence interval gives upper and lower limits of a

range within which you have a stated level of confidence that the ‘true’ average lies.

These are

commonly quoted as 95% confidence intervals, which is the level used in the benchmark reports.

They are constructed so that you can be 95% certain that the ‘true’ average lies between these

limits.

For example, chart 2, below, show a trust’s score for a question asking if respondents if staff

explained their care and treatment in a way they could understand.

Trust X has an average

score of 88, with a lower confidence limit of 85 and an upper confidence limit of 90. This means

that we can be 95% confident that the ‘true’ trust score lies between 85 and 90.

Chart 2

The width of the confidence interval gives some indication of how cautious we should be; a very

wide interval may indicate that more data should be collected before any firm conclusions are

made. In the example above, the confidence intervals are relatively small. In chart three below,

the confidence intervals are much wider: the trust has a score of 88, with a lower confidence

interval of 81 and an upper confidence interval of 95. The confidence intervals are wider here as

fewer people responded to this question (as it was only answered by respondents who were given

advice over the telephone).

Chart 3

When considering how a trust performs, it is very important to consider the confidence interval

surrounding the score.

4). Comparing scores between trusts

The confidence intervals make it possible to determine if the results from two trusts are

significantly different.

If the ranges for two trusts overlap then there is no significant difference

between the trusts: we cannot be confident that the difference in the average scores does not

simply result from random variation.

If there is no overlap in the scores of two trusts, then we can

be confident that the results for the two trusts are genuinely different.

For example, if trust A had an average score of 70 for a particular question, with a lower

confidence limit of 60 and an upper confidence limit of 80; and trust B had a score of 80, with a

lower confidence limit of 70 and an upper confidence limit of 90, then the two averages scores

are

not

significantly different as the confidence intervals overlap.

This is illustrated in table 1 (below).

By contrast, if trust C had an average score of 70 for a question, with a lower confidence limit of

66 and an upper confidence limit of 74; and trust D had a score of 80, with a lower confidence

limit of 76 and an upper confidence limit of 84, then the two scores

are

significantly different as

the confidence intervals

do not

overlap.

This is illustrated in table 1.

Tables 1 (left) and 2 (right):

Trust

qX_lcl qX_mean qX_ucl

Trust

qX_lcl qX_mean qX_ucl

60.00

70.00

80.00

66.00

70.00

74.00

70.00

80.00

90.00

76.00

80.00

84.00

Difference is

not

statistically

significant.

Difference

statistically significant.

Chart five (below) shows the ranges between the upper and lower confidence limits for all trusts

for the question asking if respondents found that the telephone advisors explained the advice in a

way they could understand. This is shown as a line for each trust that shows the area in which we

can be 95% confident that their ‘true’ score lies. As this question is only answered by those who

received advice over the telephone, the confidence intervals are quite wide for most trusts.

It can be seen that when confidence intervals are considered, the scores for many trusts do not

differ significantly from each other – each trust’s confidence interval overlaps with all others for

Q12. In other words, trusts are typically quite similar once confidence intervals are taken into

account.

Chart 5

Q12 - Did they [telephone advisor] explain the advice they gave you in a way you could understand?

Score

Naïve trust rank (ascending)

Even when the majority of respondents answer a question, it may be hard to differentiate trust

scores.

Chart 6 shows the confidence intervals for all trusts on the question asking respondents

whether they felt that overall they were treated with respect and dignity by ambulance service

staff. In this chart, we can see that the majority of trusts are very similar

once the confidence

intervals are taken into account

For example, the trust with the lowest average score for this

question is not significantly different from all other trusts.

This strongly demonstrates why the

data should not be used to construct league tables; league tables cannot fairly account for

confidence intervals and, without these, differences are implied where there are none.

Chart 6

Q30 - Overall do you feel the ambulance service staff treated you with respect and dignity?

Score

Naïve trust rank (ascending)

5.) How the data was calculated

The data in the benchmark reports is calculated by converting responses to particular questions

into scores. These were calculated by converting each respondent’s answer to a question into a

score (from 0 to 100) then averaging these to arrive at a single score for the trust, for each

question. The higher the score, the better a trust is performing. An example of a scored question

is shown below. A ‘scored’ questionnaire is available for each survey on the Care Quality

Commission website which shows how each question is scored.

Q4:

Was the ambulance control room operator reassuring?

100

Yes, definitely

Yes, to some extent

Don’t know/ Can’t remember

In most cases, the scores are allocated such that the most positive possible response

corresponds to a score of 100 and the least positive to a score of 0, with intermediary options

assigned scores at equal intervals.

Note that this approach is equivalent to that typically used

with Likert scales.

Please also note that it is not appropriate to score all questions within the questionnaire for

benchmarking purposes. This is because not all of the questions assess the trusts in any way (for

example, the question “Where were you when the ambulance service was called?”), or they may

be ‘filter questions’ designed to filter out respondents to whom following questions do not apply

(for example “Were you in any pain at the time?”). More specific reasons for not scoring some

questions for the Category C survey are provided in the benchmark reports.

Format of the Data

Results shown in the benchmark reports are based on ‘standardised’ data.

We know that the

views of a respondent can reflect not only their experience of NHS services, but can also relate to

certain demographic characteristics, such as their age and sex. For example, older respondents

tend to report more positive experiences than younger respondents, and women tend to report

less positive experiences than do men. Because the mix of patients varies across trusts (for

example, one trust may serve a considerably older population than another), this could potentially

lead to the results for a trust appearing better or worse than they would if they had a slightly

different profile of patients. To account for this we ‘standardise’ the data. Standardising data

adjusts for these differences and enables the results for trusts to be compared more fairly than

could be achieved using non-standardised data. More detailed information for each survey is

available on request to the Care Quality Commission survey team by contacting:

patient.survey@cqc.org.uk

Care Quality Commission

May 2009

Univers
Ebooks
Livres audio
Presse
Podcasts
BD
Documents

Livre audio en ligne - Développement personnel Livre en ligne Tout le catalogue Tous les Intérêts

Guide to benchmark reports

YouScribe

Le catalogue

Le service

Les conditions

Guide to benchmark reports

Attention

YouScribe

Le catalogue

Le service

Les conditions