"Transfer Learning will be the next driver of ML success"
- Andrew Ng
What actually is Transfer Learning? Let's find out !!
One of the greatest ideas in Deep Learning is that you can take knowledge that a neural network has learned in one task and use it to to apply for a separate task.
For example you made a network for detecting MRI scans and use that knowledge for detecting X-rays images, this concept is called Transfer Learning.
This is a very effective and time-saving technique for working on massive datasets with diversified applications, because the most important and time consuming part while executing a neural network is the training.
This technology helps us in not only accelerating the process but also achieving State of the Art results with minimal resource investment.
While dealing with transfer learning, we come across two most used words : Pre-trained model and current model.
Pre-trained model - It is a saved network that was previously trained on a large data-set, typically suited to your requirement. You either use the pre-trained model as is or use transfer learning to customize this model to your custom use case.
The idea behind transfer learning is that if a model is trained on a large and generic dataset, this model will effectively serve as a Base model for all the upcoming similar architectures.
Think about it in this way - Take effort to build once and Clone the design and tweak it to your liking.
You can then take advantage of these learned feature maps without having to start from scratch.
"Transfer learning is the improvement of learning in a new task through the transfer of knowledge from a related task that has already been learned."
- Handbook of Research and Machine Learning Application(2009)
When to use Transfer Learning?
For example - If you wish to apply transfer learning for prediction in Task B while transfering knowledge from Task A, then the following conditions are to be met -
- Task A and B have same input (both tasks should have same input types like:image type, audio type, etc).
- Task A should have been exposed to large volumes of Data.
- Features extracted by Task A should be relatable for Task B.
How does Transfer Learning Works ?
When working with small datasets, it is a common practice to take advantage of features learned by a model trained on a larger datasets in the same domain. This is done by instantiating the pre-trained model and adding a Fully Connected Layer - (FCN) or a Layers of your choice.
The pre-trained model is "frozen" and only the weights and biases of the classifier get updated during training.
Models which are used for performing Transfer Learning are highly trained on a large datasets and optimized to increase their productivity.
Where to Implement Transfer Learning ?
Transfer learning is an optimization algorithm, a shortcut to saving time and gaining better performance.
It is an approach to try if you can satisfy the following conditions -
- Identify a related task with abundant data and you have the resources to develop a model for that task and reuse it in your own problem
- Or there is a pre-trained model available that you can use as a starting point for your own model.
In some problems where you may not have access to large volumes of Data, transfer learning can enable you to develop Enterprise grade models that you simply would fail to otherwise.
Congratulations we successfully explored the concepts of Transfer Learning, which can accelerate the Development Life cycle for AI based solutions.
This is the future of primitive Deep Learning techniques.
STAY TUNED for more on HOW, WHERE and WHY? 😁