teaching/scientificComputing
Archived
3
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

[regression] first exercise

Browse Source
This commit is contained in:
Jan Benda
2020-12-20 23:16:56 +01:00
parent c2e4d4e40c
commit 4b18c855b9
15 changed files with 444 additions and 75 deletions
Download Patch File Download Diff File Expand all files Collapse all files

0
regression/exercises/fitting.tex → regression/exercises/exercises00-de.tex
View File

4
regression/exercises/expdecay.m Normal file
View File

@@ -0,0 +1,4 @@
function x = expdecay(t, tau)
% return the exponential function x = e^{-t/tau}
x = exp(-t./tau);
end

10
regression/exercises/expdecaydata.m Normal file
View File

@@ -0,0 +1,10 @@
tau = 10.0; % membrane time constant in ms
dt = 0.05; % sampling interval in ms
noisesd = 0.05; % measurement noise in mV
time = 0.0:dt:5*tau; % time vector
voltage = expdecay(time, tau); % exponential decay
voltage = voltage + noisesd*randn(1, length(voltage)); % plus noise
plot(time, voltage);

34
regression/exercises/expdecaydescent.m Normal file
View File

@@ -0,0 +1,34 @@
function [tau, taus, mses] = expdecaydescent(t, x, tau0, epsilon, threshold)
% Gradient descent for fitting a decaying exponential.
%
% Arguments: t, vector of time points.
% x, vector of the corresponding measured data values.
% tau0, initial value for the time constant.
% epsilon: factor multiplying the gradient.
% threshold: minimum value for gradient
%
% Returns: tau, the final value of the time constant.
% taus: vector with all the tau-values traversed.
% mses: vector with the corresponding mean squared errors
tau = tau0;
gradient = 1000.0;
taus = [];
mses = [];
count = 1;
while abs(gradient) > threshold
taus(count) = tau;
mses(count) = expdecaymse(t, x, tau);
gradient = expdecaygradient(t, x, tau);
tau = tau - epsilon * gradient;
count = count + 1;
end
end
function mse = expdecaymse(t, x, tau)
mse = mean((x - expdecay(t, tau)).^2);
end
function gradient = expdecaygradient(t, x, tau)
h = 1e-7; % stepsize for derivative
gradient = (expdecaymse(t, x, tau+h) - expdecaymse(t, x, tau))/h;
end

29
regression/exercises/expdecayplot.m Normal file
View File

@@ -0,0 +1,29 @@
expdecaydata; % generate data
tau0 = 2.0;
eps = 1.0;
thresh = 0.00001;
[tauest, taus, mses] = expdecaydescent(time, voltage, tau0, eps, thresh);
subplot(2, 2, 1); % top left panel
hold on;
plot(taus, '-o');
plot([1, length(taus)], [tau, tau], 'k'); % line indicating true tau value
hold off;
xlabel('Iteration');
ylabel('tau');
subplot(2, 2, 3); % bottom left panel
plot(mses, '-o');
xlabel('Iteration steps');
ylabel('MSE');
subplot(1, 2, 2); % right panel
hold on;
% generate x-values for plottig the fit:
tt = 0.0:0.01:max(time);
xx = expdecay(tt, tauest);
plot(time, voltage, '.'); % plot original data
plot(tt, xx, '-r'); % plot fit
xlabel('Time [ms]');
ylabel('Voltage [mV]');
legend('data', 'fit', 'location', 'northeast');
pause

116
regression/exercises/gradientdescent-1.tex
View File

@@ -1,6 +1,6 @@
\documentclass[12pt,a4paper,pdftex]{exam}
\newcommand{\exercisetopic}{Resampling}
\newcommand{\exercisetopic}{Gradient descent}
\newcommand{\exercisenum}{9}
\newcommand{\exercisedate}{December 22th, 2020}
@@ -15,67 +15,83 @@
\begin{questions}
\question We want to fit the straigth line \[ y = mx+b \] to the
data in the file \emph{lin\_regression.mat}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\question \qt{Read sections 8.1 -- 8.5 of chapter 8 on ``optimization
and gradient descent!}\vspace{-3ex}
In the lecture we already prepared the cost function
(\code{meanSquaredError()}), and the gradient
(\code{meanSquaredGradient()}) (read chapter 8 ``Optimization and
gradient descent'' in the script, in particular section 8.4 and
exercise 8.4!). With these functions in place we here want to
implement a gradient descend algorithm that finds the minimum of the
cost function and thus the slope and intercept of the straigth line
that minimizes the squared distance to the data values.
The algorithm for the descent towards the minimum of the cost
function is as follows:
\begin{enumerate}
\item Start with some arbitrary parameter values (intercept $b_0$
and slope $m_0$, $\vec p_0 = (b_0, m_0)$ for the slope and the
intercept of the straight line.
\item \label{computegradient} Compute the gradient of the cost function
at the current values of the parameters $\vec p_i$.
\item If the magnitude (length) of the gradient is smaller than some
small number, the algorithm converged close to the minimum of the
cost function and we abort the descent. Right at the minimum the
magnitude of the gradient is zero. However, since we determine
the gradient numerically, it will never be exactly zero. This is
why we just require the gradient to be sufficiently small
(e.g. \code{norm(gradient) < 0.1}).
\item \label{gradientstep} Move against the gradient by a small step
$\epsilon = 0.01$:
\[\vec p_{i+1} = \vec p_i - \epsilon \cdot \nabla f_{cost}(m_i, b_i)\]
\item Repeat steps \ref{computegradient} -- \ref{gradientstep}.
\end{enumerate}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\question \qt{Fitting the time constant of an exponential function}
Let's assume we record the membrane potential from a photoreceptor
neuron. We define the resting potential of the neuron to be at
0\,mV. By means of a brief current injection we increase the
membrane potential by exactly 1\,mV. We then record how the membrane
potential decays exponentially down to the resting potential. We are
interested in the membrane time constant and therefore want to fit
an exponential function to the recorded time course of the membrane
potential.
\begin{parts}
\part Implement the gradient descent in a function that returns
the parameter values at the minimum of the cost function and a vector
with the value of the cost function at each step of the algorithm.
\part Implement (and document!) the exponential function
\begin{equation}
\label{expfunc}
x(t) = e^{-t/\tau}
\end{equation}
with the membrane time constant $\tau$ as a matlab function
\code{expdecay(t, tau)} that takes as arguments a vector of
time points and the membrane time constant. The function returns
\eqnref{expfunc} computed for each time point as a vector.
\begin{solution}
\lstinputlisting{descent.m}
\lstinputlisting{expdecay.m}
\end{solution}
\part Plot the data and the straight line with the parameter
values that you found with the gradient descent method.
\part Plot the development of the costs as a function of the
iteration step.
\part Let's first generate the data. Set the membrane time
constant to 10\,ms. Generate a time vector with sample times
between zero and five times the membrane time constant and a
sampling interval of 0.05\,ms. Then compute a vector containing
the corresponding measurements of the membrane potential using the
\code{expdecay()} function and adding some measurement noise with
a standard deviation of 0.05\.mV (\code{randn()} function). Also
plot the data.
\begin{solution}
\lstinputlisting{descentfit.m}
\lstinputlisting{expdecaydata.m}
\end{solution}
\part For checking the gradient descend method from (a) compare
its result for slope and intercept with the position of the
minimum of the cost function that you get when computing the cost
function for many values of the slope and intercept and then using
the \code{min()} function. Vary the value of $\epsilon$ and the
minimum gradient. What are good values such that the gradient
descent gets closest to the true minimum of the cost function?
\part Implement the gradient descent algorithm for finding the
least squares for the exponential function \eqref{expfunc}. The
function takes as arguments the measured data, an initial value
for the estimation of the membrane time constant, the $\epsilon$
factor, and the threshold for the length of the gradient where to
terminate the algorithm. The function should return the estimated
membrane time constant at the minimum of the mean squared error, a
vector with the time constants, and a vector with the mean squared
errors for each step of the algorithm.
\begin{solution}
\lstinputlisting{checkdescent.m}
\lstinputlisting{expdecaydescent.m}
\end{solution}
\part Call the gradient descent function with the generated data.
Watch the value of the gradient and of tau and adapt $\epsilon$
and the threshold accordingly (they differ quite dramatically from
the ones in the script for the cubic fit).
\part Generate three plots: (i) the values of the time constant
for each iteration step, (ii) the mean squared error for each
iteration step, and (iii) the measured data and the fitted
exponential function.
\begin{solution}
\lstinputlisting{expdecayplot.m}
\end{solution}
\end{parts}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\question \qt{Read sections 8.6 -- 8.8 of chapter 8 on ``optimization
and gradient descent!}\vspace{-3ex}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\question \qt{Fitting the full exponential function}
\begin{parts}
\part Use the functions \code{polyfit()} and \code{lsqcurvefit()}
provided by matlab to find the slope and intercept of a straight
line that fits the data. Compare the resulting fit parameters of

92
regression/exercises/gradientdescent-2.tex Normal file
View File

@@ -0,0 +1,92 @@
\documentclass[12pt,a4paper,pdftex]{exam}
\newcommand{\exercisetopic}{Gradient descent}
\newcommand{\exercisenum}{9}
\newcommand{\exercisedate}{December 22th, 2020}
\input{../../exercisesheader}
\firstpagefooter{Prof. Dr. Jan Benda}{}{jan.benda@uni-tuebingen.de}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}
\input{../../exercisestitle}
\begin{questions}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\question We want to fit the straigth line \[ y = mx+b \] to the
data in the file \emph{lin\_regression.mat}.
In the lecture we already prepared the cost function
(\code{meanSquaredError()}), and the gradient
(\code{meanSquaredGradient()}) (read chapter 8 ``Optimization and
gradient descent'' in the script, in particular section 8.4 and
exercise 8.5!). With these functions in place we here want to
implement a gradient descend algorithm that finds the minimum of the
cost function and thus the slope and intercept of the straigth line
that minimizes the squared distance to the data values.
The algorithm for the descent towards the minimum of the cost
function is as follows:
\begin{enumerate}
\item Start with some arbitrary parameter values (intercept $b_0$
and slope $m_0$, $\vec p_0 = (b_0, m_0)$ for the slope and the
intercept of the straight line.
\item \label{computegradient} Compute the gradient of the cost function
at the current values of the parameters $\vec p_i$.
\item If the magnitude (length) of the gradient is smaller than some
small number, the algorithm converged close to the minimum of the
cost function and we abort the descent. Right at the minimum the
magnitude of the gradient is zero. However, since we determine
the gradient numerically, it will never be exactly zero. This is
why we just require the gradient to be sufficiently small
(e.g. \code{norm(gradient) < 0.1}).
\item \label{gradientstep} Move against the gradient by a small step
$\epsilon = 0.01$:
\[\vec p_{i+1} = \vec p_i - \epsilon \cdot \nabla f_{cost}(m_i, b_i)\]
\item Repeat steps \ref{computegradient} -- \ref{gradientstep}.
\end{enumerate}
\begin{parts}
\part Implement the gradient descent in a function that returns
the parameter values at the minimum of the cost function and a vector
with the value of the cost function at each step of the algorithm.
\begin{solution}
\lstinputlisting{descent.m}
\end{solution}
\part Plot the data and the straight line with the parameter
values that you found with the gradient descent method.
\part Plot the development of the costs as a function of the
iteration step.
\begin{solution}
\lstinputlisting{descentfit.m}
\end{solution}
\part For checking the gradient descend method from (a) compare
its result for slope and intercept with the position of the
minimum of the cost function that you get when computing the cost
function for many values of the slope and intercept and then using
the \code{min()} function. Vary the value of $\epsilon$ and the
minimum gradient. What are good values such that the gradient
descent gets closest to the true minimum of the cost function?
\begin{solution}
\lstinputlisting{checkdescent.m}
\end{solution}
\part Use the functions \code{polyfit()} and \code{lsqcurvefit()}
provided by matlab to find the slope and intercept of a straight
line that fits the data. Compare the resulting fit parameters of
those functions with the ones of your gradient descent algorithm.
\begin{solution}
\lstinputlisting{linefit.m}
\end{solution}
\end{parts}
\end{questions}
\end{document}