← Back to portfolio

arXiv 2021 · Second author · Jul 2021

Back-Translated Task Adaptive Pretraining

Role · Second author · augmentation, experiments, writing

01Background & Goals

  • A study that combines translation-based augmentation with task-adaptive pre-training to improve BERT-style models when data is scarce.

02Contributions

Augmentation and experiments
  1. Implemented back-translation (translate to a pivot language and back), with nucleus sampling for diversity
  2. Implemented and compared augmentation methods such as EDA, TF-IDF and embedding swapping
  3. Produced the augmentation seminar video and co-wrote the paper

03Tech Stack

Framework / Platform
PyTorch, Fairseq, Hydra
Methodology
Back-translation, Task-adaptive pre-training, Data augmentation

04Results

  • Released as an arXiv preprint (2021, second author)

05Reference Material