a2c [1907.00409] Evaluating Language Model Finetuning Techniques for Low-resource Languages

Enable JavaScript to see more content

arXiv2code // top new 14d 1m 2m 3m // Enable JavaScript to see more content

[share]	Go to arXiv [ De La Salle University,College of Computer Studies ]Download as Jupyter Notebook	2019-07-02
[1907.00409] Evaluating Language Model Finetuning Techniques for Low-resource Languages We show that language model finetuning methods aid in low-resource settings, especially when the number of expert-annotated examples is scarce Abstract: Unlike mainstream languages (such as English and French), low-resource languages often suffer from a lack of expert-annotated corpora and benchmark resources that make it hard to apply state-of-the-art techniques directly. In this paper, we alleviate this scarcity problem for the low-resourced Filipino language in two ways. First, we introduce a new benchmark language modeling dataset in Filipino which we call WikiText-TL-39. Second, we show that language model finetuning techniques such as BERT and ULMFiT can be used to consistently train robust classifiers in low-resource settings, experiencing at most a 0.0782 increase in validation error when the number of training examples is decreased from 10K to 1K while finetuning using a privately-held sentiment dataset. ‹ Figure 1: Overall ULMFiT pretraining and finetuning framework. An AWD-LSTM (?) is pretrained on a language modeling task. The weights are then reused with no modifications to the architecture. For finetuning, the model is first finetuned, again using language modeling, this time to the text of the target dataset to adapt to its own vocabulary and idiosyncracies. Lastly, a “classification layer” is added to the model and is finetuned for text classification. Adapted from ?. (Methodology)Figure 2: Overall BERT pretraining and finetuning framework. Note that the same architecture in pretraining is also used in finetuning with little-to-no modification in structure. After masked-language model and next-sentence prediction is pretrained, we transfer the weights of the model to downstream tasks, with question answering and entailment shown in this example. Adapted from ?. (Methodology)›

Related: TFIDF [1911.12391] SimpleBooks: Long-term dependency book dataset with simplified English vocabulary for word-level language modeling[1910.09295] Localization of Fake News Detection via Multitask Transfer Learning[1906.01496] Regularization Advantages of Multilingual Neural Language Models for Low Resource Domains[1902.10547] An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models[1905.09642] An Investigation of Transfer Learning-Based Sentiment Analysis in Japanese[1810.10222] Universal Language Model Fine-Tuning with Subword Tokenization for Polish[1909.00100] Small and Practical BERT Models for Sequence Labeling[1907.01256] A Neural Grammatical Error Correction System Built On Better Pre-training and Sequential Transfer Learning[1307.1662] Polyglot: Distributed Word Representations for Multilingual NLP[1801.06146] Universal Language Model Fine-tuning for Text Classification Mentions [1409.0473] Neural Machine Translation by Jointly Learning to Align and Translate[1810.04805] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[1806.08730] The Natural Language Decathlon: Multitask Learning as Question Answering[1708.02182] Regularizing and Optimizing LSTM Language Models Related: Semantic Math [1905.08711] Lightweight Network Architecture for Real-Time Action Recognition[1905.08711] Lightweight Network Architecture for Real-Time Action Recognition[1904.05342] ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission1806.01830[1604.07178] Weighted Spectral Cluster Ensemble[1902.10667] Bridging the Gap: Attending to Discontinuity in Identification of Multiword Expressions[1812.09489] Random Projection in Deep Neural Networks[1702.04265] Design of a Time Delay Reservoir Using Stochastic Logic: A Feasibility Study[1901.09082] A Kalman filtering induced heuristic optimization based partitional data clustering[1908.05436] Learning Trajectory Dependencies for Human Motion Prediction