a2c [1911.06171v1] Unsupervised Pre-training for Natural Language Generation: A Literature Review

Enable JavaScript to see more content

arXiv2code // top new 14d 1m 2m 3m // Enable JavaScript to see more content

[share]	Go to arXiv [ ]Download as Jupyter Notebook	2019-11-16
[1911.06171v1] Unsupervised Pre-training for Natural Language Generation: A Literature Review In this review, we survey the recent unsupervised methods to utilize large-scale corpora for NLG purposes, with a highlight on those aiming at facilitating the integration of pre-trained models with downstream tasks Abstract Recently, unsupervised pre-training is gaining increasing popularity in the realm of computational linguistics, thanks to its surprising success in advancing natural language understanding (NLU) and the potential to effectively exploit large-scale unlabelled corpus. However, regardless of the success in NLU, the power of unsupervised pre-training is only partially excavated when it comes to natural language generation (NLG). The major obstacle stems from an idiosyncratic nature of NLG: Texts are usually generated based on certain context, which may vary with the target applications. As a result, it is intractable to design a universal architecture for pre-training as in NLU scenarios. Moreover, retaining the knowledge learned from pre-training when learning on the target task is also a non-trivial problem. This review summarizes the recent efforts to enhance NLG systems with unsupervised pre-training, with a special focus on the methods to catalyse the integration of pretrained models into downstream tasks. They are classified into architecture-based methods and strategy-based methods, based on their way of handling the above obstacle. Discussions are also provided to give further insights into the relationship between these two lines of work, some informative empirical phenomenons, as well as some possible directions where future work can be devoted to. ‹ (Architecture-based Methods) (Conditional Masked Language Model)›

Related: TFIDF [1908.05672] Towards Making the Most of BERT in Neural Machine Translation[1908.08206] Denoising based Sequence-to-Sequence Pre-training for Text Generation[1905.03197v1] Unified Language Model Pre-training for Natural Language Understanding and Generation[1905.03197] Unified Language Model Pre-training for Natural Language Understanding and Generation[1905.07129] ERNIE: Enhanced Language Representation with Informative Entities[1910.07370] Evolution of transfer learning in natural language processing[1902.02671v1] BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning[1910.01462] Towards Understanding of Medical Randomized Controlled Trials by Conclusion Generation Related: Semantic Math [1903.07801] Robust Visual Tracking Using Dynamic Classifier Selection with Sparse Representation of Label Noise[1703.02185] Indoor Localization by Fusing a Group of Fingerprints Based on Random Forests[1712.04695] UV-GAN: Adversarial Facial UV Map Completion for Pose-invariant Face Recognition[1705.05085] Active Learning for Graph Embedding[1808.10551] Dynamic mode decomposition in vector-valued reproducing kernel Hilbert spaces for extracting dynamical structure among observables[1903.08375] Uncertainty quantification of molecular property prediction with Bayesian neural networks[1903.08375] Uncertainty quantification of molecular property prediction with Bayesian neural networks[1908.04355] Adversarial Neural Pruning[1907.04325] Image based Eye Gaze Tracking and its Applications[1503.02108] Maximum a Posteriori Adaptation of Network Parameters in Deep Models