Neural Networks: Chapter 2 - Fundamentals

Neural Networks Jun 9, 2021

In our previous article, we saw how we can relate Artificial Neural Networks with the Nervous system of our body and how Human Intelligence can be mapped to the computation of signals. Today we will look into some basic concepts of Neural Networks.

Single Layer Perceptron

The simplest neural network is known as Perceptron. It consists of 1 Input layer and 1 output layer. A simple instantiation of Neural Network which has a single-layer network with a set of inputs directly mapped to output by using a generalized variation of a linear function. Such a neural network is also referred to as the Perceptron.

Single Layer Perceptron

The architecture of the perceptron is shown above, in which a single input layer transmits the features to the output node. The edges from the input to the output contain the weights w1 . . . w5 with which the features are multiplied and added at the output node. Subsequently, the step function is applied to convert the aggregated value into a class label. The sign function serves the role of an activation function.

To solve a more complex problem, the number of neurons needed for a single-layer perceptron increases exponentially. To solve this we introduce depth in the network, which increases the number of hidden layers and decreases computational load and the density of neurons per layer.

Multi-Layer Perceptron

When the network has to solve a more complex equation, the single-layer perceptron does not hold well. As the complexity increase, we need to increase the number of layers in our neural network. This introduces depth in the network and proposed the term Deep Learning.

In multi-layer neural networks, the neurons are arranged in a layered fashion, in which the input and output layers are separated by a group of hidden layers. This layer-wise architecture of the neural network is also referred to as a feed-forward network.

Multi-Layer Perceptron

The network is feed with the input and all the computations are done on subsequent layers known as hidden layers. These layers apply an activation function and the results are then aggregated at a final layer, also called the output layer.

Training a Neural Network

Now that we have seen the basic architecture of a Neural Network, let's talk about how to train it. There are various frameworks available in the market like Tensorflow, Pytorch, etc. which makes it simple to define and train a neural network. But it is important to understand how the training process works and what is the mathematics behind it.

In Supervised learning problems, we have the target variable with us, and all we need to do is update the weights and biases of the network in such a way that the input passed to the network gives us the desired output. In other words, we need to minimize the error between the expected output and the observed output. This difference can be mathematically depicted as a Loss Function.

The goal of the training process is to minimize the Loss function.

Two major parameters drive the training process, i.e. weights and biases. Once the network can understand the right combination of weights and biases for every layer, it is considered to be trained.

The choice of Activation functions, Loss functions, and Optimizers needs to be made from a list of options available. The "One Size Fits All" concept doesn't work here. The choice of the parameters depends on the problem statement.

Forward Pass

In this pass, the input for training is fed into the neural network. This results in a forward cascade of computations across the layers, using the current set of weights. The final predicted output can be compared to that of the training instance and the derivative of the loss function concerning the output is computed. The derivative of this loss now needs to be computed concerning the weights in all layers in the Backward pass.

Backward Pass

The main goal of the Backward Pass is to learn the gradient of the loss function concerning the different weights by using the chain rule of differential calculus. These gradients are used to update the weights. Since these gradients are learned in the backward direction, starting from the output node, this learning process is referred to as the Backward Pass or Backpropagation.

Training process


We have seen all the fundamental units required in building and training a Neural Network.

This was a high-level explanation of basic concepts. In future posts, we will deep dive into all the important units that are required in Neural Networks. Feel free to read about these concepts and STAY TUNED for articles covering advanced concepts of Neural Networks.


Arpit Jain

Machine Learning Engineer

Great! You've successfully subscribed.
Great! Next, complete checkout for full access.
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.