Impact of Deep Learning on Transfer Learning : A Review (original) (raw)

A Survey on Deep Transfer Learning

Artificial Neural Networks and Machine Learning – ICANN 2018, 2018

As a new classification platform, deep learning has recently received increasing attention from researchers and has been successfully applied to many domains. In some domains, like bioinformatics and robotics, it is very difficult to construct a large-scale well-annotated dataset due to the expense of data acquisition and costly annotation, which limits its development. Transfer learning relaxes the hypothesis that the training data must be independent and identically distributed (i.i.d.) with the test data, which motivates us to use transfer learning to solve the problem of insufficient training data. This survey focuses on reviewing the current researches of transfer learning by using deep neural network and its applications. We defined deep transfer learning, category and review the recent research works based on the techniques used in deep transfer learning.

A Review of Deep Transfer Learning and Recent Advancements

ArXiv, 2022

A successful deep learning model is dependent on extensive training data and processing power and time (known as training costs). There exist many tasks without enough number of labeled data to train a deep learning model. Further, the demand is rising for running deep learning models on edge devices with limited processing capacity and training time. Deep transfer learning (DTL) methods are the answer to tackle such limitations, e.g., fine-tuning a pre-trained model on a massive semi-related dataset proved to be a simple and effective method for many problems. DTLs handle limited target data concerns as well as drastically reduce the training costs. In this paper, the definition and taxonomy of deep transfer learning is reviewed. Then we focus on the subcategory of network-based DTLs since it is the most common types of DTLs that have been applied to various applications in the last decade.

Transfer Learning in Deep Neural Networks

Transfer learning (TL), have been becoming extremely popular due to its appalling property of being able to work at a different domain, where it's not trained. Often training at such domain is very costly due to various reasons such as lack of sufficient data etc. Recently Deep Neural Network (DNN) based models have been widely used in the context of TL. These networks as clear transferable layers representing common latent features, hence intuitive to transfer. In this paper we survey various DNN architectures, different training methods adopted to train those and their performance in several application domains.

Unsupervised and transfer learning challenge: a deep learning approach

Unsupervised and Transfer Learning Workshop, in conjunction with ICML, 2011

Learning good representations from a large set of unlabeled data is a particularly challenging task. Recent work (see Bengio (2009) for a review) shows that training deep architectures is a good way to extract such representations, by extracting and disentangling gradually higher-level factors of variation characterizing the input distribution. In this paper, we describe different kinds of layers we trained for learning representations in the setting of the Unsupervised and Transfer Learning Challenge. The strategy of our team won the ...

Improving transfer learning accuracy by reusing stacked denoising autoencoders

Transfer learning is a process that allows reusing a learning machine trained on a problem to solve a new problem. Transfer learning studies on shallow architectures show low performance as they are generally based on hand-crafted features obtained from experts. It is therefore interesting to study transference on deep architectures, known to directly extract the features from the input data. A Stacked Denoising Autoencoder (SDA) is a deep model able to represent the hierarchical features needed for solving classification problems. In this paper we study the performance of SDAs trained on one problem and reused to solve a different problem not only with different distribution but also with a different tasks. We propose two different approaches: 1) unsupervised feature transference, and 2) supervised feature transference using deep transfer learning. We show that SDAs using the unsupervised feature transference outperform randomly initialized machines on a new problem. We achieved 7% relative improvement on average error rate and 41% on average computation time to classify typed uppercase letters. In the case of supervised feature transference, we achieved 5.7% relative improvement in the average error rate, by reusing the first and second hidden layer, and 8.5% relative improvement for the average error rate and 54% speed up w.r.t the baseline by reusing all three hidden layers for the same data. We also explore transfer learning between geometrical shapes and canonical shapes, we achieved 7.4% relative improvement on average error rate in case of supervised feature transference approach.

Supervised Representation Learning: Transfer Learning with Deep Autoencoders

2015

Transfer learning has attracted a lot of attention in the past decade. One crucial research issue in transfer learning is how to find a good representation for instances of different domains such that the divergence between domains can be reduced with the new representation. Recently, deep learning has been proposed to learn more robust or higherlevel features for transfer learning. However, to the best of our knowledge, most of the previous approaches neither minimize the difference between domains explicitly nor encode label information in learning the representation. In this paper, we propose a supervised representation learning method based on deep autoencoders for transfer learning. The proposed deep autoencoder consists of two encoding layers: an embedding layer and a label encoding layer. In the embedding layer, the distance in distributions of the embedded instances between the source and target domains is minimized in terms of KL-Divergence. In the label encoding layer, lab...

AutoTune: Automatically Tuning Convolutional Neural Networks for Improved Transfer Learning

Neural Networks, 2021

Transfer learning enables solving a specific task having limited data by using the pre-trained deep networks trained on large-scale datasets. Typically, while transferring the learned knowledge from source task to the target task, the last few layers are fine-tuned (re-trained) over the target dataset. However, these layers are originally designed for the source task which might not be suitable for the target task. In this paper, we introduce a mechanism for automatically tuning the Convolutional Neural Networks (CNN) for improved transfer learning. The CNN layers are tuned with the knowledge from target data using Bayesian Optimization. Initially, we train the final layer of the base CNN model by replacing the number of neurons in the softmax layer with the number of classes involved in the target task. Next, the CNN is tuned automatically by observing the classification performance on the validation data (greedy criteria). To evaluate the performance of the proposed method, experiments are conducted on three benchmark datasets, e.g., CalTech-101, CalTech-256, and Stanford Dogs. The classification results obtained through the proposed AutoTune method outperforms the standard baseline transfer learning methods over the three datasets by achieving 95.92%, 86.54%, and 84.67% accuracy over CalTech-101, CalTech-256, and Stanford Dogs, respectively. The experimental results obtained in this study depict that tuning of the pre-trained CNN layers with the knowledge from the target dataset confesses better transfer learning ability.

Instance-Based Deep Transfer Learning

2019 IEEE Winter Conference on Applications of Computer Vision (WACV), 2019

Deep transfer learning recently has acquired significant research interest. It makes use of pre-trained models that are learned from a source domain, and utilizes these models for the tasks in a target domain. Model-based deep transfer learning is probably the most frequently used method. However, very little research work has been devoted to enhancing deep transfer learning by focusing on the influence of data. In this paper, we propose an instance-based approach to improve deep transfer learning in a target domain. Specifically, we choose a pre-trained model from a source domain and apply this model to estimate the influence of training samples in a target domain. Then we optimize the training data of the target domain by removing the training samples that will lower the performance of the pre-trained model. We later either fine-tune the pre-trained model with the optimized training data in the target domain, or build a new model which is initialized partially based on the pre-trained model, and fine-tune it with the optimized training data in the target domain. Using this approach, transfer learning can help deep learning models to capture more useful features. Extensive experiments demonstrate the effectiveness of our approach on boosting the quality of deep learning models for some common computer vision tasks, such as image classification.

Image Classification Using Transfer Learning and Deep Learning

International Journal of Engineering and Computer Science, 2021

Deep learning models have demonstrated improved efficacy in image classification since the ImageNet Large Scale Visual Recognition Challenge started since 2010. Classification of images has further augmented in the field of computer vision with the dawn of transfer learning. To train a model on huge dataset demands huge computational resources and add a lot of cost to learning. Transfer learning allows to reduce on cost of learning and also help avoid reinventing the wheel. There are several pretrained models like VGG16, VGG19, ResNet50, Inceptionv3, EfficientNet etc which are widely used. This paper demonstrates image classification using pretrained deep neural network model VGG16 which is trained on images from ImageNet dataset. After obtaining the convolutional base model, a new deep neural network model is built on top of it for image classification based on fully connected network. This classifier will use features extracted from the convolutional base model.

Unsupervised and transfer learning challenge

The 2011 International Joint Conference on Neural Networks, 2011

We organized a data mining challenge in "unsupervised and transfer learning" (the UTL challenge), in collaboration with the DARPA Deep Learning program. The goal of this year's challenge was to learn good data representations that can be re-used across tasks by building models that capture regularities of the input space. The representations provided by the participants were evaluated by the organizers on supervised learning "target tasks", which were unknown to the participants. In a first phase of the challenge, the competitors were given only unlabeled data to learn their data representation. In a second phase of the challenge, the competitors were also provided with a limited amount of labeled data from "source tasks", distinct from the "target tasks". We made available large datasets from various application domains: handwriting recognition, image recognition, video processing, text processing, and ecology. The results indicate that learned data representation yield results significantly better than what can be achieved with raw data or data preprocessed with standard normalizations and functional transforms. The UTL challenge is part of the IJCNN 2011 competition program 1. The website of the challenge remains open for submission of new methods beyond the termination of the challenge as a resource for students and researchers 2 .