Improving cross-lingual language understanding with consistency regularization-based fine-tuning

Abstract

Fine-tuning pre-trained cross-lingual language models alleviates the need for annotated data in different languages, as it allows the models to transfer task-specific supervision between languages, especially from high- to low-resource languages. In this work, we propose to improve cross-lingual language understanding with consistency regularization-based fine-tuning. Specifically, we use example consistency regularization to penalize the prediction sensitivity to four types of data augmentations, i.e., subword sampling, Gaussian noise, code-switch substitution, and machine translation. In addition, we employ model consistency to regularize the models trained with two augmented versions of the same training set. Experimental results on the XTREME benchmark show that our method (the code is available at https://github.com/bozheng-hit/xTune)  achieves significant improvements across various cross-lingual language understanding tasks, including text classification, question answering, and sequence labeling. Furthermore, we extend our method to the few-shot cross-lingual transfer setting, particularly considering a more realistic setting where machine translation systems are available. Meanwhile, machine translation as data augmentation can be well combined with our consistency regularization method. Experimental results demonstrate that our method also benefits the few-shot scenario.

Appendices

Appendix 1: Statistics of XTREME datasets

See Table 9.

Table 9 Statistics for the datasets in the XTREME benchmark. we report the number of training examples (#Train), and the number of languages (#Lang)

Appendix 2: Hyper-parameters

See Table 10.

Table 10 The best hyper-parameters used for xTune under the cross-lingual transfer setting

See Table 11.

Table 11 The best hyper-parameters used for xTune under the translate-train-all setting

1.1 Conventional cross-lingual fine-tuning

For XNLI, PAWS-X, POS and NER, we fine-tune 10 epochs. For XQuAD and MLQA, we fine-tune 4 epochs. For TyDiQA-GoldP, we fine-tune 20 epochs and 10 epochs for base and large model, respectively. We select $\lambda _{1}$ in [1.0, 2.0, 5.0], $\lambda _{2}$ in [0.3, 0.5, 1.0, 2.0, 5.0]. For learning rate, we select in [5e-6, 7e-6, 1e-5, 1.5e$-$5] for large models, [7e-6, 1e-5, 2e-5, 3e-5] for base models. We use batch size 32 for all datasets and 10% of total training steps for warmup with a linear learning rate schedule. Our experiments are conducted with a single 32GB Nvidia V100 GPU, and we use gradient accumulation for large-size models. The other hyper-parameters for the two-stage xTune training are shown in Tables 10 and 11.

1.2 Few-shot cross-lingual fine-tuning

During the source-training stage, we use the same hyper-parameters as conventional cross-lingual fine-tuning. During the target-adapting stage, for POS and NER tasks, we fine-tune the model 100 epochs and select the model on the development set every epoch, with an early-stop** strategy of 10 epochs. For MLQA and XQuAD tasks, we fine-tune the model for 2 or 3 epochs and use the model of the last epoch. For learning rate, we select in [2e-5, 1e-5]. We use batch size 32 for POS and NER tasks and batch size 8 for XQuAD and MLQA tasks. We set $\lambda _1$ to 5.0 in UDA since it has the best performance in the previous experiments.

Appendix 3: Results for each dataset and language

We provide detailed results for each dataset and language below. We compare our method against $\text {XLM-R}_\text {large}$ for cross-lingual transfer setting, FILTER [15] for translate-train-all setting.

Appendix 4: How to select data augmentation strategies in xTune

We give instructions on selecting a proper data augmentation strategy depending on the corresponding task.

1.1 Classification

The two distribution in example consistency $\mathcal {R}_{1}$ can always be aligned. Therefore, we recommend using machine translation as data augmentation if the machine translation systems are available. Otherwise, the priority of our data augmentation strategies is code-switch substitution, subword sampling and Gaussian noise.

1.2 Span extraction

The two distribution in example consistency $\mathcal {R}_{1}$ can not be aligned in translation-pairs. Therefore, it is impossible to use machine translation as data augmentation in example consistency $\mathcal {R}_{1}$. We prefer to use code-switch when applying example consistency $\mathcal {R}_{1}$ individually. However, when the training corpus is augmented with translations, since the bilingual dictionaries between arbitrary language pairs may not be available, we recommend using subword sampling in example consistency $\mathcal {R}_1$.

1.3 Sequence labeling

Similar to span extraction, the two distribution in example consistency $\mathcal {R}_{1}$ can not be aligned in translation-pairs. Therefore, we do not use machine translation in example consistency $\mathcal {R}_{1}$. Unlike classification and span extraction, sequence labeling requires finer-grained information and is more sensitive to noise. We found code-switch is worse than subword sampling as data augmentation in both example consistency $\mathcal {R}_{1}$ and model consistency $\mathcal {R}_{2}$, it will even degrade performance for certain hyper-parameters. Thus we recommend using subword sampling in example consistency $\mathcal {R}_{1}$, and use machine translation to augment the English training corpus if machine translation systems are available, otherwise subword sampling.

Appendix 5: Results for each language

See Tables 12, 13, 14, 15, 16 and 17.

Table 12 PAWSX results (accuracy scores) for each language

Table 13 XQuAD results (F1/EM scores) for each language

Table 14 MLQA results (F1/EM scores) for each language

Table 15 TyDiQA-GolP results (F1/EM scores) for each language

Table 16 POS results (accuracy) for each language

Table 17 NER results (F1 scores) for each language