Fundamental error in "Trends in serious head injuries..." Cook and



J

James Annan

Guest
I sent the following to "Injury prevention on-line" over a
week ago, but it shows no sign of being published and my follow-
up email has not been answered. I guess someone might as
well see it, typo and all. Isn't the internet great for
vanity publishing?

Neither Cook nor Sheikh replied to my email, either.

Their original article, to which this refers, can be found
on

http://ip.bmjjournals.com/cgi/content/full/9/3/266

-------------------------------------------------------
----------
"Fundamental error in "Trends in serious head injuries..."
Cook and Sheikh 2003"
-------------------------------------------------------
----------

<!-- article ID: 9/3/266 -->

<P> The main conclusion of Cook and Sheikh (2003), that a
bicycle helmet prevents 60% of head injuries, is
incorrect due to a fundamental error in the way they
have treated their percentages. A correct analysis
demonstrates unequivocally that there must be major
confounding factors in their data set that they have
failed to take into account, and therefore any estimate
of helmet effectiveness is purely speculative.

<Q> Assuming that their basic analysis of the data is
correct (although the numbers they quote in the text do
not actually appear to match the figure plotted), they
arrive at a figure of a 3.6% for the reduction in the
head injury (HI) rate for cyclists, over and above the
"background" reduction that pedestrians have also seen.
They assume that this drop in HI is due to increased
helmet-wearing. However, this reduction is presented in
terms of the number of percentage points, and relative
to the baseline value of 27.9% HI for cyclists in 1995-
6 it actually represents a
3.6/27.9 = 13% drop in the HI rate.

<P> The decrease in the number of helmetless cyclists over
the same interval is 5.8 percentage points from a
baseline of 84% unhelmeted, giving the percentage drop
as 5.8/84 = 7%. Cook and Sheikh calculate helmet
effectiveness to be given by the ratio 3.6/5.8 = 60%.
However the correct expression to use is 13/7 = 186%.
In other words, "helmet effectiveness" is so high that
each helmet does not just save its wearer, but a non-
wearer too. At this rate, head injuries would be
eliminated completely if just a little over half of all
cyclists wore them! This is clearly ludicrous.

<Q> A more reasonable conclusion to draw from this would be
that there are some other factors that are responsible
for the large drop in HI rate, and therefore any
attempt to attribute some part of the total 30%
(8.49/27.9) change to the provably marginal impact of a very
small number of extra helmet wearers is at best
highly speculative and fraught with inaccuracy.

<P> What makes this all the more poignant is the fact that
the authors have recently produced a book entitled
"Basic skills in statistics"!

<Q> James Annan
 
D

Dorre

Guest
James Annan <[email protected]> wrote in message news:<[email protected]>...
> I sent the following to "Injury prevention on-line" over a
> week ago, but it shows no sign of being published and my
> follow-up email has not been answered. I guess someone
> might as well see it, typo and all. Isn't the internet
> great for vanity publishing?

Sometimes these things take a little longer. Have you
checked whether you are in excess of the formal word limit?

The editor may also think that the nature of your
criticism means he should seek advice before publishing
it. What you say is, of course, correct. I was very
suspicious when I first saw that claim and thought I'd
better write it out formally to check, but (shame on me!)
I never got round to it.

Below is a more formal way of expressing the same thing. I
didn't bother sending it to Injury Prevention. But if James
or anyone else thinks they can make use of it, feel free!

An alternative, more mathematical/formal way of making the
same points as James
------------------------------------------------------------
----------------

There is a miscalculation in Cook and Sheikh's paper.[1]

From 1995/6 to 2000/1 the percentage of hospital admissions
with head injury (%HI) fell from 27.9% to 20.4% for cyclists
and 26.9% to 22.8% for pedestrians, declines of 7.5 and 4.1
percentage points.

Cook and Sheikh claimed that, because percent helmet wearing
(%HW) increased by 5.8% (from 16.0% to 21.8%) and %HI of
cyclists fell by
3.6% more than pedestrians, helmets must prevent 3.6/5.8=
60% of head injuries.

The above argument confuses percentages with percentage
points. A decline from 27.9 to 20.4 represents a fall of
27%. The predicted fall in %HI of 3.6 percentage points more
than pedestrians (25.9% to
4.3%, after subtracting pedestrian trends) represents a
13.9% drop in numbers of head injuries. Could such a
relatively large fall be caused by an increase of just 5.8
percentage points in %HW?

A mathematical approach is needed to provide the answer.[2]
If h and n are the probabilities of head injury for helmet
wearers and non-wearers respectively, then: %HI = %HW*h +
(100-%HW)*n (1) The predicted value of %HI at the midpoint
of Cook and Sheikh's data is 24.1%; with %HW of 18.9%. If
helmets prevent 60% of head injuries, h = 0.4n, so, from
equation (1), n = 0.272.

Also from equation (1) with n = 0.272, increased %HW
from 16.0% to
5.8% should decrease %HI from 24.59 to 23.64, 0.95
percentage points. This is nowhere near the 3.6 percentage
points reported by Cook and Sheikh. Even if helmets
prevented 100% of head injuries, %HI would fall by only
1.7 percentage points.

Cook and Sheikh's estimate of 60% is therefore invalid.
Increased helmet wearing cannot explain the larger drop in
%HI of cyclists compared to pedestrians. Other factors
must also have been involved, such as gradual changes in
the age composition of cyclists, or the relative amounts
of on vs off-road cycling. Cook and Sheikh's data
therefore provide no real evidence of reduced HI from
increased helmet wearing.

In Australia and New Zealand, helmet laws increased %HW
dramatically, in many cases from less than 30% to more than
80% of all cyclists in less than a year, yet there were no
large or obvious corresponding changes in %HI over and above
prevailing trends. In contrast to Cook and Sheikh's
analysis, this tells us a great deal about the benefits of
helmet laws.[3]

References
6. Cook A, Sheikh A. Trends in serious head injuries
among English cyclists and pedestrians. Inj Prev
2003; 9: 266–267.
7. Robinson DL. Head injuries and bicycle helmet laws.
Accid Anal Prev 1996; 28: 463-475.
8. Robinson DL. Reasons for trends in cyclist injury
data. Injury Prevention 2004; 10: 126–127.
 
J

James Annan

Guest
[email protected] (Dorre) wrote in message news:<973d9a9.0406101[email protected]>...
> James Annan <[email protected]> wrote in
> message news:<[email protected]
> net.or.jp>...
> > I sent the following to "Injury prevention on-line" over
> > a week ago, but it shows no sign of being published and
> > my follow-up email has not been answered. I guess
> > someone might as well see it, typo and all. Isn't the
> > internet great for vanity publishing?
>
> Sometimes these things take a little longer. Have
> you checked whether you are in excess of the formal
> word limit?

There's not a hard limit, it took some searching to find out
that I might be slightly in excess of the guideline figure
but I can't imagine them agonising over that detail without
telling me.

> The editor may also think that the nature of your
> criticism means he should seek advice before
> publishing it.

Well, it is possible, but a bit rude to not only not tell me
of this off his own bat (once he had exceeded his own time
limit) but to ignore my email asking what was going on.
Also, IP have a deliberate policy of wading in to
controversial areas, and my letter is a lot more polite and
less controversial than several others they have recently
published ("Injury Prevention injured itself by publishing
such unprofessional work. A retraction is warranted, with
support for this methodology and its spurious conclusions
disavowed.")

OTOH, a couple of years ago Science refused to publish a
straightforward correction to one of the most famous papers
measuring global warming over recent decades (it
overestimated the trend by about 10% due to a numerical
error). It continues to be cited regularly. So nothing
really surprises me any more...

James
 
S

Simon Brooke

Guest
in message <[email protected]>, Dorre
('[email protected]') wrote:

> James Annan <[email protected]> wrote in
> message news:<[email protected]
> net.or.jp>...
>> I sent the following to "Injury prevention on-line" over
>> a week ago, but it shows no sign of being published and
>> my follow-up email has not been answered. I guess someone
>> might as well see it, typo and all. Isn't the internet
>> great for vanity publishing?
>
> Sometimes these things take a little longer. Have
> you checked whether you are in excess of the formal
> word limit?

[snip]

> Below is a more formal way of expressing the same thing.
> I didn't bother sending it to Injury Prevention. But if
> James or anyone else thinks they can make use of it,
> feel free!
>
> An alternative, more mathematical/formal way of making the
> same points as James
>
------------------------------------------------------------
----------------
>
> There is a miscalculation in Cook and Sheikh's paper.[1]

[snip: excellent presentation]

Perhaps it would be a good thing to send this one in also,
so that each could reinforce the other?

--
[email protected] (Simon Brooke)
http://www.jasmine.org.uk/~simon/

;; no eternal reward will forgive us now for wasting the
dawn. ;; Jim Morrison
 
D

David Martin

Guest
On 11/6/04 10:24 pm, in article
[email protected], "James Annan"
<[email protected]> wrote:

> Simon Brooke wrote:
>
>
>> Perhaps it would be a good thing to send this one in
>> also, so that each could reinforce the other?
>
> It appeared at last:
>
> http://ip.bmjjournals.com/cgi/eletters/9/3/266#59
>

Well done!

And as it is linked from the original paper, everyone will
see it (a great improvement over dead tree journals).

If I were the authors I would be very embarrassed.

..d
 
D

Dave Kahn

Guest
On Sat, 12 Jun 2004 20:04:04 +0100, David Martin
<[email protected]> wrote:

>And as it is linked from the original paper, everyone will
>see it (a great improvement over dead tree journals).
>
>If I were the authors I would be very embarrassed.

Unfortunately it won't undo the damage already done by the
stories in the general press.

--
Dave...

Get a bicycle. You will not regret it. If you live. -
Mark Twain
 
A

Anonymous Cowar

Guest
On Sat, 12 Jun 2004 20:04:04 +0100, David Martin wrote:

> On 11/6/04 10:24 pm, in article [email protected]
> net.or.jp, "James Annan"
> <[email protected]> wrote:
>
>> Simon Brooke wrote:
>>
>>
>>> Perhaps it would be a good thing to send this one in
>>> also, so that each could reinforce the other?
>>
>> It appeared at last:
>>
>> http://ip.bmjjournals.com/cgi/eletters/9/3/266#59
>>
>
> Well done!
>
> And as it is linked from the original paper, everyone will
> see it (a great improvement over dead tree journals).
>
> If I were the authors I would be very embarrassed.

I'm embarrassed for posting the link - nice one!

AC
 
T

Tony Raven

Guest
Dave Kahn wrote:
> On Sat, 12 Jun 2004 20:04:04 +0100, David Martin
> <[email protected]> wrote:
>
>> And as it is linked from the original paper, everyone
>> will see it (a great improvement over dead tree
>> journals).
>>
>> If I were the authors I would be very embarrassed.
>
> Unfortunately it won't undo the damage already done by the
> stories in the general press.

The recent press has been on their more recent paper anyway.
Need to get some responses up to that.

Tony
 
J

James Annan

Guest
Tony Raven wrote:

> The recent press has been on their more recent paper
> anyway. Need to get some responses up to that.

AIUI, at least one has been sent (not by me), pointing out
the shaky foundations of their research (including, but not
limited to, this error). Not that the press will be
interested, of course.

James
 
J

Just Zis Guy

Guest
On Sun, 13 Jun 2004 15:52:21 +0900, James Annan
<[email protected]> wrote in message
<[email protected]>:

>AIUI, at least one has been sent (not by me), pointing out
>the shaky foundations of their research (including, but not
>limited to, this error). Not that the press will be
>interested, of course.

By whom? Malcolm's latest rebuttal is of the same
paper as yours.

Guy
--
May contain traces of irony. Contents liable to settle after posting.
http://www.chapmancentral.co.uk

88% of helmet statistics are made up, 65% of them at Washington University
 
J

James Annan

Guest
Just zis Guy, you know? wrote:
> On Sun, 13 Jun 2004 15:52:21 +0900, James Annan
> <[email protected]> wrote in message <[email protected]
> net.or.jp>:
>
>
>>AIUI, at least one has been sent (not by me), pointing out
>>the shaky foundations of their research (including, but
>>not limited to, this error). Not that the press will be
>>interested, of course.
>
>
> By whom? Malcolm's latest rebuttal is of the same paper
> as yours.

No, it (unless you are talking about something else again)
is actually sent to JRSM in reply to Sheikh's "Unethical
arguments" paper which appeared in that journal. Which,
since it leans substantially on this incorrect analysis, is
equally undermined by the error (and weak for a number of
other reasons too).

James
 
J

Just Zis Guy

Guest
On Sun, 13 Jun 2004 18:45:20 +0900, James Annan
<[email protected]> wrote in message
<[email protected]>:

>> By whom? Malcolm's latest rebuttal is of the same paper
>> as yours.

>No, it (unless you are talking about something else again)
>is actually sent to JRSM in reply to Sheikh's "Unethical
>arguments" paper which appeared in that journal.

I may be looking at a different draft from Malcolm.

Guy
--
May contain traces of irony. Contents liable to settle after posting.
http://www.chapmancentral.co.uk

88% of helmet statistics are made up, 65% of them at Washington University