Towards Understanding Transfer Learning with Applications to Medical Imaging

In almost all applications of deep learning, transfer learning -- where a deep neural network is pretrained on a first task, and then
finetuned on a different, target task -- plays a crucial role. This is especially the case in medical imaging applications, where the de-facto method is to use a standard large model from natural image datasets (e.g. ImageNet) and corresponding pretrained weights. However, the effects of transfer learning are poorly understood, with several recent papers in the natural image setting challenging commonly believed intuitions. For medical applications, fundamental differences between data and task specifications mean that these questions and many others remain unexplored. In this talk, I survey some of the recent results on understanding transfer learning in natural images, as well as findings on transfer learning for medical tasks. In this latter setting, we observe a number of counter-intuitive results, including connections between feature reuse and overparametrization, surprisingly strong performance of lightweight non-standard architectures, and even feature independent effects of transfer.