# Understanding Neural Network Foundations: Perceptron, ADALINE and MLP

Perceptron and ADALINE are often considered as prototypes of deep learning; which is important to understand architecture of neural network.

### Perceptron

**Perceptron (or Binary Classifier)** is an algorithm that classifies data into two categories by determining which class an input belongs to. Unlike regression models which focus on prediction and analysis, the perceptron specializes in classifying inputs into binary classes represented as $$0, 1$$.

{% hint style="info" %}
**History of Perceptron**

**Perceptron** was invented by Warren McCulloch and Walter Pitts in 1943. Rather than software, it was implemented as a machine—the IBM 704—designed to classify images. Later, the Mark 1 Perceptron emerged as the software version of this system.
{% endhint %}

#### Formulation and Learning Rule

It does not use partial derivatives, but instead relies on the simple linear separability of the data.

$$
f(z) = f\[w(t) \cdot x\_j]
$$

1. Initialize weights to $$0$$ or a random number.
2. Calculate outputs: $$y\_j(t) = f\[w(t) \cdot x\_j]$$
3. Update weights: $$w\_i(t + 1) = w\_i(t) + r \cdot (d\_j - y\_j(t)) \cdot x\_{j, i}$$

#### Comparing Perceptron and Logistic Regression

The perceptron is often confused with logistic regression. Though they share similarities, they are entirely different concepts.

* Perceptron outputs a hard class label based on a threshold, while logistic regression outputs a probability using the sigmoid function.
* Logistic regression is a probabilistic model, while the perceptron is deterministic.
* The perceptron uses hinge loss instead of gradient descent rules and cross-entropy.
* The perceptron converges only when data is linearly separable, while logistic regression always converges.

### ADALINE

**ADALINE (Adaptive Linear Neuron, Adaptive Linear Element)** is an enhanced version of the perceptron. It classifies data using a layer of parallel perceptrons. This structure serves as a prototype for artificial neural networks.

#### Formulation and Learning Rule

It takes multiple inputs and produce a single output as a multi-layer neural network composed of various nodes.

$$
y = \sum^{n}\_{j=0}x\_jw\_j + \theta
$$

<details>

<summary><strong>Term Definitions</strong></summary>

* $$x$$ represents the input vector while $$x\_0 = 1$$ is bias.
* $$y$$ represents the model's output.
* $$w$$ represents the weights, where $$w\_0 = 0$$ is used for local bias.
* $$n$$ is the number of inputs in the dataset.
* $$\theta$$ represents the global bias constant.

</details>

* The least mean square error is calculated as $$E = (o - y)^2$$.
* Update the weights: $$w \leftarrow w + \eta(o-y) \cdot x$$.
  * $$\eta$$ represents the learning rate.
  * $$o$$ represents the target output value.

{% hint style="info" %}
**MADALINE (Many ADALINE)**, a variant of ADALINE, uses a structure that connects three ADALINE units linearly. It is similar to modern neural networks but differs in that it uses different functions per layer, making backpropagation impossible.
{% endhint %}

### MLP, Multi-Layer Perceptron

**Multi-Layer Perceptron ( or MLP, fully connected artificial neural network)** consists of three layers of perceptrons with non-linear activation functions. Its primary purpose is to classify inputs in a linear manner.

* Every perceptron in an MLP is fully connected to the next layer of perceptrons, giving each perceptron multidimensional weights $w\_{i,j}$.
* These perceptrons function as signal processing units, similar to neurons in the human brain, which is why they are called **neurons**.
* Every neuron has an activation function that maps scalar responses to a non-linear number range. This is a crucial concept in MLPs that enables their functionality and improves their performance.
  * In most implementations, MLPs use the hyperbolic tangent or sigmoid function as their activation function.

{% hint style="info" %}
It is considered a very prototype of modern neural networks and is sometimes called a *“vanilla neural network”*, though it differs in its use of forward propagation.
{% endhint %}

#### Formulation and Learning Rule

The learning rule is based on the concept of a neuron; it updates the neuron's weights by calculating partial derivatives of their cost/loss function. This represents a fundamental mechanism of modern deep learning.

$$
w\_j(l)(n) = \sum\_{i} w\_{ji}(l)(n)⋅y\_i(l−1)(n)+b\_j(l)(n)y\_j(l)(n)=\phi\large(w\_j(l)(n)\large)y\_j(l)(n)=\phi \large(w\_j(l)(n)\large)
$$

* Calculate the error $$\epsilon$$ at the output layer using $$\epsilon\_j(n) = d\_j(n) - y\_j(n)$$, where $$d\_j(n)$$ represents the desired output values of the model.
  * $$\therefore e(n) = \frac{1}{2} \cdot \sum {e\_j^2(n)}$$
* Update using the gradient descent rule: $$\Delta w\_{ji}(n) = \eta \cdot \frac{\Delta \epsilon(n)}{\Delta w\_{j}(n)} \cdot y\_i(n)$$.
  * $$e\_j$$ represents the output of layer $$j$$.
  * $$y\_i(n)$$ represents the output of neuron i in the previous layer.
  * $$w\_i$$ represents the weights of neuron$$\_i$$, while $$\eta$$ is the learning rate.
  * $$\frac{\Delta \epsilon(n)}{\Delta w\_i(n)}$$ is the partial derivative of $$\epsilon$$ with respect to $$w\_i$$, which can be expressed as: $$\frac{d\epsilon(n)}{dv\_j(n)}=e\_j(n)\phi'(w\_j(n)) = \phi'(w\_j(n)) \cdot \sum\_{k}-\frac{d\epsilon(n)}{dv\_k(n)}w\_{kj}(n)$$ where $$\phi$$ represents the activation function.

{% hint style="info" %}
Most commonly, mean square error serves as $$\epsilon$$, the cost/loss function.
{% endhint %}
