← Back to portfolio

JKIIE 2021 · First author · Dec 2021

Multi-modal Korean Emotion Recognition with Consistency Regularization

Role · First author · model design, experiments, writing

01Background & Goals

  • We studied a multimodal framework that adds an external speech-recognition module to improve speech-based emotion recognition.
  • Real-world speech recognition makes mistakes, so we needed a training method robust to them.

02Contributions

Multimodal framework and robust training
  1. Designed a multimodal architecture over speech and STT text (BERT, Wav2vec 2.0)
  2. Designed a consistency-regularization loss that aligns the prediction distributions of the gold-text and STT paths

03Tech Stack

Framework / Platform
PyTorch, Fairseq, Hydra
Methodology
Multimodal learning, Self-supervised learning, Consistency regularization

04Results

  • Published in JKIIE 47(6) (first author); code released
  • Best Master's Thesis Award, KIIE Master's Thesis Competition (Nov 2021)

05Reference Material