The course slides (Deck 2,
Slide 93
and
Slides 112–121)
derives three different asymptotic
confidence intervals for binomial data
and mentions an exact
one (that involves heavy
computing and is not derived). This web page examines the exact performance
of all four (plots coverage as a function of the parameter value).
As discussed in the course slides (Deck 2,
Slide 92)
the coverage probability considered as a function of the parameter can never
be a constant function when the data have a discrete distribution. So even
the exact
confidence interval does not achieve exactly the nominal
coverage for all parameter values but only at least the nominal
coverage, so is perhaps better termed conservative-exact
.
Usual Binomial Confidence Intervals
The usual
confidence interval is the one shown on
Slide 113, Deck 2
of the course slides. It is the one taught in all intro stats courses.
The following R code plots the coverage as a function of the parameter. The solid line is the graph of this function (the vertical sections are not, strictly speaking, part of the graph, but it is easiest to plot it this way). The dashed horizontal line is the nominal coverage level.
Score Binomial Confidence Intervals
The score
confidence interval is the one shown on
Slide 116, Deck 2
of the course slides. It is the computed by the R function
prop.test
(on-line
help).
This one is beginning to be taught in some intro stats courses.
The following R code plots the coverage as a function of the parameter. The solid line is the graph of this function (the vertical sections are not, strictly speaking, part of the graph, but it is easiest to plot it this way). The dashed horizontal line is the nominal coverage level.
The warnings emitted by the prop.test are about the hypothesis
test not the confidence interval and are irrelevant.
Variance Stabilized Binomial Confidence Intervals
The confidence interval using the variance stabilizing transformation is the one shown on Slide 120, Deck 2 of the course slides.
The following R code plots the coverage as a function of the parameter. The solid line is the graph of this function (the vertical sections are not, strictly speaking, part of the graph, but it is easiest to plot it this way). The dashed horizontal line is the nominal coverage level.
Clopper-Pearson Binomial Confidence Intervals
The Clopper-Pearson confidence interval is not derived on the course
slides. It is the computed by the R function
binom.test
(on-line
help).
The following R code plots the coverage as a function of the parameter. The solid line is the graph of this function (the vertical sections are not, strictly speaking, part of the graph, but it is easiest to plot it this way). The dashed horizontal line is the nominal coverage level.
Modified Usual Binomial Confidence Intervals
Geyer (2009, Electronic Journal of Statistics 3, 259 289) proposed a simple modification that fixes the behavior of binomial confidence intervals near zero and one (and also applies to other distributions). For the binomial distribution it says that when we observe zero successes, the confidence interval should be from zero to 1 − α1 ⁄ n, where coverage 1 − α is wanted and n is the sample size. And is says that when we observe all successes (n out of n), the confidence interval should be from α1 ⁄ n to one.
In this section we use this modification when zero or n successes
are observed and use the usual
confidence interval (section
Usual Binomial Confidence Intervals above)
for other data.
The following R code plots the coverage as a function of the parameter. The solid line is the graph of this function (the vertical sections are not, strictly speaking, part of the graph, but it is easiest to plot it this way). The dashed horizontal line is the nominal coverage level.
Modified Variance Stabilized Binomial Confidence Intervals
We use the same modification used in the preceding section but now apply it to the Variance Stabilized Binomial Confidence Intervals described above.
The following R code plots the coverage as a function of the parameter. The solid line is the graph of this function (the vertical sections are not, strictly speaking, part of the graph, but it is easiest to plot it this way). The dashed horizontal line is the nominal coverage level.
Summary
Of the three confidence intervals with simple definitions, only the
score intervals behave well for all values of the
unknown parameter. But the usual
intervals
and the variance-stabilized intervals can
be modified by patching up what they do when x = 0 or
x = n is observed, as described in the sections
Modified Usual Binomial Confidence Intervals
and
Modified Variance Stabilized Binomial
Confidence Intervals above.
The Clopper-Pearson Binomial Confidence Intervals, of course, also have good performance, because they are defined to be that way (guaranteed conservative).
Similar sorts of plots and similar sorts of modifications can be cooked up for other confidence intervals, but we do not belabor the subject and leave it here.
