Transfer Learning for Speech and Language Processing
\authorblockNDong Wang and Thomas Fang Zheng \authorblockA1. Center for Speech and Language Technologies (CSLT) Research Institute of Information Technology, Tsinghua University 2. Tsinghua National Lab for Information Science and Technology Beijing, 100084, P.R.China
Abstract
Transfer learning is a vital technique that generalizes models trained for one setting or task to other settings or tasks. For example in speech recognition, an acoustic model trained for one language can be used to recognize speech in another language, with little or no re-training data. Transfer learning is closely related to multi-task learning (cross-lingual vs. multilingual), and is traditionally studied in the name of ‘model adaptation’. Recent advance in deep learning shows that transfer learning becomes much easier and more effective with high-level abstract features learned by deep models, and the ‘transfer’ can be conducted not only between data distributions and data types, but also between model structures (e.g., shallow nets and deep nets) or even model types (e.g., Bayesian models and neural models). This review paper summarizes some recent prominent research towards this direction, particularly for speech and language processing. We also report some results from our group and highlight the potential of this very interesting research field111This survey will be continuously updated online () to reflect the recent progress on transfer learning. .
中文速览
迁移学习（transfer learning）在语音和语言处理中面临的核心挑战是：人类语言极度多样且资源分布严重不均，绝大多数语言和领域的标注数据严重匮乏，导致单一来源训练的模型难以直接推广使用。本文系统梳理了深度学习时代迁移学习的主要方法，涵盖模型自适应、异构迁移、多任务学习、领域自适应以及模型结构之间的迁移（如浅层网络向深层网络的知识蒸馏），并重点介绍了这些方法在语音识别和文档分类两个任务上的具体应用与作者团队的最新实验结果。研究表明，深度模型学到的高层抽象表示极大地降低了跨语言、跨领域乃至跨模型类型迁移的难度，使"用少量目标域数据甚至零数据实现有效迁移"成为可能。这项工作的意义在于，它为散落在各个子领域、各自为战的迁移学习研究提供了统一的分类框架，帮助研究者看清自己方法的定位、局限与改进方向，对资源稀缺语言和新兴领域的实用系统开发具有直接的指导价值。
原文 arXiv:1511.06066;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1511.06066v1