Bootstrapping Regression Models
Appendix to An R and S-PLUS Companion to Applied Regression

John Fox
January 2002

1

Basic Ideas

Bootstrapping is a general approach to statistical inference based on building a sampling distribution for a statistic by resampling from the data at hand. The term ‘bootstrapping,’ due to Efron (1979), is an allusion to the expression ‘pulling oneself up by one’s bootstraps’ – in this case, using the sample data as a population from which repeated samples are drawn. At first blush, the approach seems circular, but has been shown to be sound.
Two S libraries for bootstrapping are associated with extensive treatments of the subject: Efron and
Tibshirani’s (1993) bootstrap library, and Davison and Hinkley’s (1997) boot library. Of the two, boot, programmed by A. J. Canty, is somewhat more capable, and will be used for the examples in this appendix.
There are several forms of the bootstrap, and, additionally, several other resampling methods that are related to it, such as jackknifing, cross-validation, randomization tests, and permutation tests. I will stress the nonparametric bootstrap.
Suppose that we draw a sample S = {X1 , X2 , ..., Xn } from a population P = {x1 , x2 , ..., xN }; imagine further, at least for the time being, that N is very much larger than n, and that S is either a simple random sample or an independent random sample from P;1 I will briefly consider other sampling schemes at the end of the appendix. It will also help initially to think of the elements of the population (and, hence, of the sample) as scalar values, but they could just as easily be vectors (i.e., multivariate).
Now suppose that we are interested in some statistic T = t(S) as an estimate of the corresponding population parameter θ = t(P). Again, θ could be a vector of parameters and T the corresponding vector of estimates, but