← Back to portfolio

SK Telecom · Apr 2024 – Dec 2024

A-dot Main Agent

Role · Agent design, prompt engineering, training-data design, fine-tuning

60→80%function-call successacross 17 functions, after fine-tuning and prompt work

01Background & Goals

  • A-dot's flagship LLM agent, offering everyday conversation plus 17 functions such as exchange rates, weather, time, directions, news and subway congestion.

02Key Challenges

  • C1 Function-call accuracy and hallucination
  • C2 API cost and latency
  • C3 Multi-turn conversation quality

03Contributions

Agent design for accuracy and UI/UX integrationC1
  1. Consolidated similar functions to simplify function selection
  2. Let the conversation use in-app context such as the current playlist and UI screen
  3. Split functions with low argument accuracy into sequential calls
Prompt engineering for cost and answer accuracyC1C2
  1. Kept prompts minimal to cut API cost and used fine-tuning to instill answer style
  2. Switched argument extraction for hallucination-prone time functions to NER
Fine-tuning and evaluationC3
  1. Defined the training-data format and generated multi-turn dialogue data
  2. Fine-tuned GPT models on Azure OpenAI
  3. Ran quantitative evaluation (function selection and accuracy, argument selection and extraction) and qualitative evaluation (multi-turn fluency, accuracy of function-result answers, handling of sensitive and inappropriate requests)

04Tech Stack

Framework / Platform
Azure OpenAI
Methodology
Prompt engineering, Function calling, Fine-tuning, NER

05Results

  • Raised function-call success across 17 functions from 60% to 80% through fine-tuning and prompt engineering
  • Reduced hallucination and improved time-parsing accuracy by extracting time arguments with NER and analyzing them sequentially

06Reference Material