abstract

investigate self-tarining ans another method to utilize additional data on the same setup

3 insights

1. stronger data augmentation diminish the value of pre-training even for self-supervised learning
2. self-training is always helpful despite stronger data augmenetation
3. self-training improves on pre-training

introduction & related work

最近self−監督学習のような予備訓練方法論が登場した.
しかし,実際には,実験で強力な増強機能が用いられたり,labelデータが増えたりすると,無駄な結果が生じる.
self-trainingはこれより役に立つかもしれません
selfトレーニングの拡張性(伸縮性)と一般性(汎用性)について議論します.

Methodology

比較集団軍を3つに捕まえる.

Data Augmenetaion : flipcrop, autoaugment, randaugment

pre-training including self-supervised : efficientNet with noisy student method

self-training : pseudo labels and human labels jointly

experiment

pre-training hurts performance when stronger data augmentation is used.

more labels data diminishes the value of pre-training

self-training helps in high data/strong augmentation regimes, even when pre-training hurts

self-training works across dataset size and is additive to pre-training

self-supervised pre-training also hurts when self-training helps in high data/strong augmenetation regimes

Discussion

pre-training is not ware of the task of interest and can fail to adapt, otherwise self-training with supervised learning is more adpative to the task of interset(jointly training)

予備訓練方法論では、ラベルを多くつけるとかえって性能が損なわれると考えられていますが、self-training pseudo labelは役に立ちます(追加)

結論
制限:self-trainingはfinetunningよりも多くのリソースを必要とします.
利点:自己トレーニングの拡張性、汎用性、柔軟性

loss normalization of self-training