209 lines
10 KiB
TeX
209 lines
10 KiB
TeX
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\chapter{Bootstrap methods}
|
|
\label{bootstrapchapter}
|
|
\exercisechapter{Bootstrap methods}
|
|
|
|
Bootstrapping methods are applied to create distributions of
|
|
statistical measures via resampling of a sample. Bootstrapping offers several
|
|
advantages:
|
|
\begin{itemize}
|
|
\item Fewer assumptions (e.g. a measured sample does not need to be
|
|
normally distributed).
|
|
\item Increased precision as compared to classical methods. %such as?
|
|
\item General applicability: the bootstrapping methods are very
|
|
similar for different statistics and there is no need to specialize
|
|
the method to specific statistic measures.
|
|
\end{itemize}
|
|
|
|
\begin{figure}[tp]
|
|
\includegraphics[width=0.8\textwidth]{2012-10-29_16-26-05_771}\\[2ex]
|
|
\includegraphics[width=0.8\textwidth]{2012-10-29_16-41-39_523}\\[2ex]
|
|
\includegraphics[width=0.8\textwidth]{2012-10-29_16-29-35_312}
|
|
\titlecaption{\label{statisticalpopulationfig} Why can't we measure
|
|
properties of the full population but only draw samples?}{}
|
|
\end{figure}
|
|
|
|
Reminder: in statistics we are interested in properties of a
|
|
\enterm{statistical population} (\determ{Grundgesamtheit}), e.g. the
|
|
average length of all pickles (\figref{statisticalpopulationfig}). But
|
|
we cannot measure the lengths of all pickles in the
|
|
population. Rather, we draw samples (\enterm{simple random sample}
|
|
\enterm[SRS|see{simple random sample}]{SRS}, \determ{Stichprobe}). We
|
|
then estimate a statistical measure of interest (e.g. the average
|
|
length of the pickles) within this sample and hope that it is a good
|
|
approximation of the unknown and immeasurable true average length of
|
|
the population (\entermde{Populationsparameter}{population
|
|
parameter}). We apply statistical methods to find out how precise
|
|
this approximation is.
|
|
|
|
If we could draw a large number of simple random samples we
|
|
could calculate the statistical measure of interest for each sample
|
|
and estimate its probability distribution using a histogram. This
|
|
distribution is called the \enterm{sampling distribution}
|
|
(\determ{Stichprobenverteilung},
|
|
\subfigref{bootstrapsamplingdistributionfig}{a}).
|
|
|
|
\begin{figure}[tp]
|
|
\includegraphics[height=0.2\textheight]{srs1}\\[2ex]
|
|
\includegraphics[height=0.2\textheight]{srs2}\\[2ex]
|
|
\includegraphics[height=0.2\textheight]{srs3}
|
|
\titlecaption{\label{bootstrapsamplingdistributionfig}Bootstrapping
|
|
the sampling distribution.}{(a) Simple random samples (SRS) are
|
|
drawn from a statistical population with an unknown population
|
|
parameter (e.g. the average $\mu$). The statistical measure (the
|
|
estimation of $\bar x$) is calculated for each sample. The
|
|
measured values originate from the sampling distribution. Often
|
|
only a single random sample is drawn! (b) By applying assumption
|
|
and theories one can guess the sampling distribution without
|
|
actually measuring it. (c) Alternatively, one can generate many
|
|
bootstrap-samples from the same SRS (resampling) and use these to
|
|
estimate the sampling distribution empirically. From Hesterberg et
|
|
al. 2003, Bootstrap Methods and Permutation Tests}
|
|
\end{figure}
|
|
|
|
Commonly, there will be only a single SRS. In such cases we make use
|
|
of certain assumptions (e.g. we assume a normal distribution) that
|
|
allow us to infer the precision of our estimation based on the
|
|
SRS. For example the formula $\sigma/\sqrt{n}$ gives the standard
|
|
error of the mean which is the standard deviation of the sampling
|
|
distribution of average values around the true mean of the population
|
|
(\subfigref{bootstrapsamplingdistributionfig}{b}).
|
|
|
|
Alternatively, we can use \enterm{bootstrapping}
|
|
(\determ[Bootstrap!Verfahren]{Bootstrapverfahren}) to generate new
|
|
samples from one set of measurements
|
|
(\entermde{Resampling}{resampling}). From these bootstrapped samples
|
|
we compute the desired statistical measure and estimate their
|
|
distribution (\entermde{Bootstrap!Verteilung}{bootstrap distribution},
|
|
\subfigref{bootstrapsamplingdistributionfig}{c}). Interestingly, this
|
|
distribution is very similar to the sampling distribution regarding
|
|
its width. The only difference is that the bootstrapped values are
|
|
distributed around the measure of the original sample and not the one
|
|
of the statistical population. We can use the bootstrap distribution
|
|
to draw conclusion regarding the precision of our estimation (e.g.
|
|
standard errors and confidence intervals).
|
|
|
|
Bootstrapping methods create bootstrapped samples from a SRS by
|
|
resampling. The bootstrapped samples are used to estimate the sampling
|
|
distribution of a statistical measure. The bootstrapped samples have
|
|
the same size as the original sample and are created by randomly
|
|
drawing with replacement. That is, each value of the original sample
|
|
can occur once, multiple time, or not at all in a bootstrapped
|
|
sample. This can be implemented by generating random indices into the
|
|
data set using the \code{randi()} function.
|
|
|
|
|
|
\section{Bootstrap of the standard error}
|
|
|
|
Bootstrapping can be nicely illustrated at the example of the
|
|
\enterm{standard error} of the mean (\determ{Standardfehler}). The
|
|
arithmetic mean is calculated for a simple random sample. The standard
|
|
error of the mean is the standard deviation of the expected
|
|
distribution of mean values around the mean of the statistical
|
|
population.
|
|
|
|
\begin{figure}[tp]
|
|
\includegraphics[width=1\textwidth]{bootstrapsem}
|
|
\titlecaption{\label{bootstrapsemfig}Bootstrapping the standard
|
|
error of the mean.}{The --- usually unknown --- sampling
|
|
distribution of the mean is distributed around the true mean of
|
|
the statistical population ($\mu=0$, red). The bootstrap
|
|
distribution of the means computed from many bootstrapped samples
|
|
has the same shape as the sampling distribution but is centered
|
|
around the mean of the SRS used for resampling. The standard
|
|
deviation of the bootstrap distribution (blue) is an estimator for
|
|
the standard error of the mean.}
|
|
\end{figure}
|
|
|
|
Via bootstrapping we create a distribution of the mean values
|
|
(\figref{bootstrapsemfig}) and the standard deviation of this
|
|
distribution is the standard error of the mean.
|
|
|
|
\pagebreak[4]
|
|
\begin{exercise}{bootstrapsem.m}{bootstrapsem.out}
|
|
Create the distribution of mean values from bootstrapped samples
|
|
resampled from a single SRS. Use this distribution to estimate the
|
|
standard error of the mean.
|
|
\begin{enumerate}
|
|
\item Draw 1000 normally distributed random number and calculate the
|
|
mean, the standard deviation, and the standard error
|
|
($\sigma/\sqrt{n}$).
|
|
\item Resample the data 1000 times (randomly draw and replace) and calculate
|
|
the mean of each bootstrapped sample.
|
|
\item Plot a histogram of the respective distribution and calculate its mean and
|
|
standard deviation. Compare with the
|
|
original values based on the statistical population.
|
|
\end{enumerate}
|
|
\end{exercise}
|
|
|
|
|
|
\section{Permutation tests}
|
|
Statistical tests ask for the probability of a measured value to
|
|
originate from a null hypothesis. Is this probability smaller than the
|
|
desired \entermde{Signifikanz}{significance level}, the
|
|
\entermde{Nullhypothese}{null hypothesis} may be rejected.
|
|
|
|
Traditionally, such probabilities are taken from theoretical
|
|
distributions which are based on assumptions about the data. Thus the
|
|
applied statistical test has to be appropriate for the type of
|
|
data. An alternative approach is to calculate the probability density
|
|
of the null hypothesis directly from the data itself. To do this, we
|
|
need to resample the data according to the null hypothesis from the
|
|
SRS. By such permutation operations we destroy the feature of interest
|
|
while we conserve all other statistical properties of the data.
|
|
|
|
\begin{figure}[tp]
|
|
\includegraphics[width=1\textwidth]{permutecorrelation}
|
|
\titlecaption{\label{permutecorrelationfig}Permutation test for
|
|
correlations.}{Let the correlation coefficient of a dataset with
|
|
200 samples be $\rho=0.21$. The distribution of the null
|
|
hypothesis (yellow), optained from the correlation coefficients of
|
|
permuted and therefore uncorrelated datasets is centered around
|
|
zero. The measured correlation coefficient is larger than the
|
|
95\,\% percentile of the null hypothesis. The null hypothesis may
|
|
thus be rejected and the measured correlation is considered
|
|
statistically significant.}
|
|
\end{figure}
|
|
|
|
A good example for the application of a
|
|
\entermde{Permutationstest}{permutaion test} is the statistical
|
|
assessment of \entermde[correlation]{Korrelation}{correlations}. Given
|
|
are measured pairs of data points $(x_i, y_i)$. By calculating the
|
|
\entermde[correlation!correlation
|
|
coefficient]{Korrelationskoeffizient}{correlation
|
|
coefficient} we can quantify how strongly $y$ depends on $x$. The
|
|
correlation coefficient alone, however, does not tell whether the
|
|
correlation is significantly different from a random correlation. The
|
|
\entermde{Nullhypothese}{null hypothesis} for such a situation is that
|
|
$y$ does not depend on $x$. In order to perform a permutation test, we
|
|
need to destroy the correlation by permuting the $(x_i, y_i)$ pairs,
|
|
i.e. we rearrange the $x_i$ and $y_i$ values in a random
|
|
fashion. Generating many sets of random pairs and computing the
|
|
resulting correlation coefficients yields a distribution of
|
|
correlation coefficients that result randomly from uncorrelated
|
|
data. By comparing the actually measured correlation coefficient with
|
|
this distribution we can directly assess the significance of the
|
|
correlation (figure\,\ref{permutecorrelationfig}).
|
|
|
|
\begin{exercise}{correlationsignificance.m}{correlationsignificance.out}
|
|
Estimate the statistical significance of a correlation coefficient.
|
|
\begin{enumerate}
|
|
\item Create pairs of $(x_i, y_i)$ values. Randomly choose $x$-values
|
|
and calculate the respective $y$-values according to $y_i =0.2 \cdot x_i + u_i$
|
|
where $u_i$ is a random number drawn from a normal distribution.
|
|
\item Calculate the correlation coefficient.
|
|
\item Generate the distribution of the null hypothesis by generating
|
|
uncorrelated pairs. For this permute $x$- and $y$-values
|
|
\matlabfun{randperm()} 1000 times and calculate for each permutation
|
|
the correlation coefficient.
|
|
\item Read out the 95\,\% percentile from the resulting distribution
|
|
of the null hypothesis and compare it with the correlation
|
|
coefficient computed from the original data.
|
|
\end{enumerate}
|
|
\end{exercise}
|
|
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\printsolutions
|