CSC 446 Note: Lecture 6

Lam Tran

February 9, 2012

Multi Layer Perceptron

In a two layer neural network, the input, hidden, and output variables are represented by nodes, and the weight parameters are represented by links between the nodes. The arrow of the links indicate the direction of information flow through the network during forward propagation. The overall two stage network function takes the form

t_{k}(x, w) = σ (∑_{j=0} w_{jk} g (∑_{i=0} w_{ij} x_{i})) ,  (1)

where

z_{j} = g (∑_{i=0} w_{ij} x_{i})

are known as the hidden units, w_{ij} are the weights, x_{i} are the input variables, and the σ and g are activation functions. Non-linear functions are usually chosen for activation functions such as tanh and sigmoid functions.

Training A Network: Error Backpropagation

Given a training set of input vector {x_{i}} and its target vector {y_{i}} for i = 1 … N, we want to minimize the error function

E^{n}(w) = \frac{1}{2} ∑_{i=1}^{N} ∥ t_{i}^{n} − y_{i}^{n} ∥^{2}= ∑_{i=1}^{N} E_{i}^{n}.  (2)

The response function t_{i}^{n} take take form of (1) but to keep notation uncluttered we will omit the input parameters. To minimize the error function of a no hidden layer network, we apply the chain rule for the partial derivatives with respect to (w.r.t.) the weights w_{ij} to get

\frac{∂E^{n}}{∂w_{ij}} = ∑_{j′} \frac{∂E^{n}}{∂t_{j′}^{n}} \frac{∂t_{j′}^{n}}{∂w_{ij}}

= (t_{j}^{n} − y_{j}^{n}) \frac{∂t^{n}}{∂a_{j}} \frac{∂a_{j}}{∂w_{ij}}

= (t_{j}^{n} − y_{j}^{n}) g′(a_{j})x_{i}

= \underbrace{(t_{j}^{n} − y_{j}^{n}) g′(a_{j})}_{δ_{j}} x_{i}

= δ_{j}x_{i}  (3)

where

a_{j} = ∑_{i} w_{ij} x_{i}

is the sum of the weighted input feeds. The derivative respect to the matrix W is

\frac{∂E^{n}}{∂W} = [\frac{∂E^{n}}{∂w_{ij}} ]_{ij} = [δ_{j}x_{i}]_{ij} = δx^{T} .

We now focus on training on a neural network with a hidden layer (multi layer) that takes the same form as (2) and minimize the error w.r.t. the weights w_{ij} in this form

\frac{∂E^{n}}{∂w_{ij}} = ∑_{k} \frac{∂E^{n}}{∂t_{k}^{n}} \frac{∂t_{k}^{n}}{∂w_{ij}} = ∑_{k} \frac{∂E^{n}}{∂t_{k}^{n}} \frac{∂t_{k}^{n}}{∂a_{k}} \frac{∂a_{k}}{∂w_{ij}}

= ∑_{k}(t_{k} − y_{k}) (g′(a_{k}) \frac{∂}{∂w_{ij}} ∑_{j′} w_{j′k} g(a_{j′}))

= ∑_{k}(t_{k} − y_{k}) (g′(a_{k})w_{jk} \frac{∂}{∂w_{ij}} g(a_{j}))

= ∑_{k}(t_{k} − y_{k}) (g′(a_{k})w_{jk} g′(a_{j})x_{i})

= ∑_{k} \underbrace{(t_{k} − y_{k})g′(a_{k})w_{jk} g′(a_{j})}_{δ_{j}} x_{i}

= δ_{j}x_{i}

The derivatives with respect to the first and second layer weights are given by

\frac{∂E^{n}}{∂w_{ij}} = δ_{j}x_{i}  and  \frac{∂E_{n}}{∂w_{jk}} = δ_{k}z_{j}.

Thus the training algorithm consists of calculating weight for n Calculate weight for n

W+ = η \frac{∂E^{n}}{∂W} .