← Back to portfolio

SK Telecom · May 2022 – Dec 2023

Cupid: Answer Recommendation for Similar Questions

Role · Model development, recommendation and evaluation pipelines

16→31%AI auto-answer rateduplicate questions, Aug 1 → Aug 21, 2022
38→57%positive feedback"helpful" ratio on AI answers

01Background & Goals

  • Cupid is a community service where nearby users ask and answer each other's questions.
  • We wanted an AI feature that finds answers already given to similar questions.

02Key Challenges

  • C1 Question–answer matching accuracy across topics and regions
  • C2 Automated evaluation of the main models

03Contributions

Content-based recommendation pipelineC1
  1. Classification and extraction: extracts location and category from the question (BERT-based classifiers and NER)
  2. Filtering: narrows candidates by location, category and stop words
  3. Encoding: masks place names and converts the question to a vector with the embedding model
  4. Retrieval: finds similar questions in Elasticsearch with BM25 + cosine similarity
  5. Re-ranking: scores question containment and question–answer consistency and returns the top N
Models behind the pipelineC1
  1. Embedding model: trained a BERT-based model with a Siamese network so similar sentences sit close together
  2. Hierarchical multi-class classifier: built hierarchical topic data and trained a BERT-based classifier
  3. NER model: trained with BIO tagging to extract places and keywords
Model evaluation pipelineC2
  1. Sent items whose category changed plus N sampled items to an external vendor for ground-truth labeling
  2. Evaluated with F1 and accuracy and extracted items with low prediction accuracy
  3. Gated deployment on a golden-set evaluation after retraining
Batch processing of new dataC1
  1. Automated an Airflow batch that encodes new question–answer pairs and re-indexes the vector DB so they become recommendation candidates

04Tech Stack

Framework / Platform
PyTorch, Transformers, Elasticsearch, Airflow
Methodology
Siamese network, Hierarchical multi-class classification, BIO NER

05Results

  • AI auto-answer rate for duplicate questions rose from 16% to 31% (Aug 1 → Aug 21, 2022)
  • Positive ("helpful") feedback on AI answers rose from 38% to 57%

06Reference Material