Architecture and Learning process in neural network (original) (raw)

Last Updated : 13 May, 2026

Neural networks are a core part of machine learning that learn patterns from data to make predictions. Inspired by the human brain, they use interconnected neurons arranged in layers to process information efficiently.

Architectures of Neural Network

Neural network architecture defines how neurons are arranged and connected to process data and learn patterns. It uses multiple layers where data flows forward through weighted connections and activation functions to produce outputs.

1. Single-layer Feed Forward Network

A single-layer feed forward network is the simplest type of neural network where data moves directly from input to output without any hidden layers. It is suitable for simple, linearly separable problems.

111

Single-layer Feed Forward Network

2. Multi-layer Feed Forward Network

A multi-layer feed forward network extends basic neural networks by adding one or more hidden layers between input and output, allowing it to learn complex non-linear patterns.

222

Multi-layer Feed Forward Network

3. Competitive Network

A competitive network is a type of neural network where output neurons compete with each other to respond to an input. It uses unsupervised learning to discover patterns and group similar data.

3

Competitive Network

4. Recurrent Network

A recurrent neural network is designed to work with sequential data by using feedback connections, allowing it to remember past information and model time-based patterns.

4

Recurrent Neural Network

Learning Process in Artificial Neural Networks

The learning process in an Artificial Neural Network involves training the model to learn patterns from data by adjusting its parameters to reduce error and improve accuracy.

**1. Number of Layers in the Network

The number of layers in a neural network determines its ability to learn patterns, ranging from simple to complex relationships in data.

**2. Direction of Signal Flow

The direction of signal flow defines how information moves through the network and how past information is used.

**3. Number of Nodes in Layers

The number of nodes in each layer determines the network’s capacity to learn and represent data.

**4. Weights of Interconnected Nodes

Weights define the strength of connections between neurons and are key to the learning process.

Backpropagation in ANN

Backpropagation is a learning process in neural networks that reduces prediction error by updating weights based on the difference between actual and predicted outputs. It uses gradient descent to optimize these weights efficiently.

**Phases of Backpropagation

Backpropagation works in two main phases to train a neural network by first generating outputs and then correcting errors.

**1. Forward Phase

In the forward phase the input signals propagate from the input layer to the output layer through one or more hidden layers. During this process

**2. Backward Phase

In this phase, the error is calculated and propagated backward to update weights.

666

Backpropagation

Multi-Layer Feedforward Neural Network

Let's Consider a multi-layer feedforward neural network

555

Multi-layer feedforward neural network

1. Hidden Layer Computation

The net input to the k-th hidden neuron is given by

y_{in_k} = x_0 w_{0k} + \sum_{i=1}^{m} x_i w_{ik}

Here x_{0}=1 denotes the bias input

The output of the k-th hidden neuron is obtained by applying the hidden layer activation function f_{y}() to the net input:

y_{out_k} = f_y(y_{in_k})

where

2. Output Layer Computation

The net input to the k-th output neuron is obtained by summing the weighted outputs of all hidden layer neurons and adding a bias term. The bias input allows the model to shift the activation function and improves learning flexibility.

z_{in_k} = w'_{0k} + \sum_{i=1}^{n} y_{out_i} w'_{ik}

where

The final output of the k^{th} output neuron

z_{out_k} = f_z(z_{in_k})

This step produces the network’s predicted output for the given input

3. Error (Cost) Function

Let t_{k} be the target output of the k^{th} output neuron. The cost function defined as the sum of squared errors is:

E = \frac{1}{2} \sum_{k=1}^{n} (t_k - z_{out_{k}})^2

Since the output neuron applies an activation function f_{z}() the error can also be written as:

E = \frac{1}{2} \sum_{k=1}^{n} (t_k - f_z(z_{in_k}))^2

4. Weight Update for Hidden-to-Output Layer

To update the weights using gradient descent, the partial derivative of the error with respect to the weight w'_{jk} (connecting hidden neuron j to output neuron k) is computed as:

\frac{\partial E}{\partial w'_{jk}} = -(t_k - z_{out_k}) \cdot f'_z(z_{in_k}) \cdot y_{out_j}

5. Weight and Bias Update Equations

Using learning rate \alpha the weight update rule is:

w'_{0k}(\text{new}) = w'_{0k}(\text{old}) - \alpha \frac{\partial E}{\partial w'_{0k}}

For the bias weight:

w'_{0k}(\text{new}) = w'_{0k}(\text{old}) -\alpha.\frac{\partial E}{\partial w'_{jk}}

6. Weight Update for Input-to-Hidden Layer

The weights connecting the input layer to the hidden layer are updated using the gradient descent method. To determine how each weight should be changed, we compute the gradient of the error function with respect to the weight using the chain rule.

For weights change in weight is given by:

\Delta w_{ij} = -\alpha \cdot \frac{\partial E}{\partial w_{ij}}

Using the chain rule this gradient can be expanded as:

\frac{\partial E}{\partial w_{ij}} = \frac{\partial E}{\partial y_{out_j}} \cdot \frac{\partial y_{out_j}}{\partial y_{in_j}} \cdot \frac{\partial y_{in_j}}{\partial w_{ij}}

weight update:

w_{ij}{(\text{new})} = w_{ij}{(\text{old})} -\alpha. \frac{\partial E}{\partial w_{ij}}

Applications

Advantages

Limitations