Viva Republica Inc. (Toss) · AI Product Teamonlyl4youu@gmail.comlinkedin.com/in/joungheekimgithub.com/JoungheeKimjoungheekim.github.io/portfolio
Summary
AI Engineer who automates judgment work with LLM agents. At Toss, built a browser-use merchant review agent that saved 20 FTE in card-company review, and an inspection automation platform in which AI refines the prompts, now used by 10+ departments. At SK Telecom, built A-dot's scheduling agent (handling 67% of schedule registrations) plus movie-booking and main agents. Background in speech/NLP research (Interspeech 2022) and financial AI.
I build agents that take over review work people used to do by hand. Merchant review automation saved 20 FTE, and the inspection automation platform, where AI takes over prompt refinement, has been adopted by 10+ departments.
Viva Republica Inc. (Toss)AI Product Team · AI Engineer · Dec 2025 – Present
ad-copy typo policy pilot · mean gain of the refinement and held-out sets (36 ads)
T-01Viva Republica (Toss)May 2026 – Present
Merchant Review Automation
Automated card-company merchant review and Toss Pay merchant review with a browser-use agent that browses merchant websites like a human reviewer and collects the evidence a review needs.
When a merchant URL comes in, the browser-use review agent opens a browser and goes to the merchant website.
02 / 06
Look and reason like a reviewer
It reads the screenshot and page text together and decides where to look next based on what the review still lacks.
03 / 06
One action at a time
It runs one action — navigate, click, scroll or extract — and records what it saw and why.
04 / 06
Only what the review needs
It repeats observe → reason → act until it finds what the review needs: terms, payment methods, business details. For card-company review it follows the payment path and captures each screen.
05 / 06
Stops on its own
Once it has the public information it needs, it stops. It reads public pages one at a time, so a merchant server sees no more load than a single visitor.
06 / 06
Handed over for review
The captured screens and text are handed over as review material. Automating card-company merchant review saved 20 FTE.
Reconstructed screen · synthetic data
agent.review(merchant_url)
●example-shop.testbrowser-use
Daily CraftShopStorySupport
Handmade ceramicsSpring collection
Moon cup₩38,000
Linen tray₩24,000
Clay vase₩52,000
Terms of service
Article 12 (Returns)Buyers may return items within 7 days of receipt; refunds are processed within 3 business days after the return is confirmed.
Moon cup₩38,000Shipping ₩3,000 · free over ₩50,000Buy now
PaymentCardTransferEasy pay
Order form
Moon cup × 1₩38,000
Shipping
Payment
Credit card
Bank transfer
Card payment
Card ACard BCard CCard DCard ECard F
Total₩41,000
Pay (not requested)
AI
01 · A merchant URL to review — When a merchant URL comes in, the browser-use review agent opens a browser and goes to the merchant website.
02 · Look and reason like a reviewer — It reads the screenshot and page text together and decides where to look next based on what the review still lacks.
03 · One action at a time — It runs one action — navigate, click, scroll or extract — and records what it saw and why.
04 · Only what the review needs — It repeats observe → reason → act until it finds what the review needs: terms, payment methods, business details. For card-company review it follows the payment path and captures each screen.
05 · Stops on its own — Once it has the public information it needs, it stops. It reads public pages one at a time, so a merchant server sees no more load than a single visitor.
06 · Handed over for review — The captured screens and text are handed over as review material. Automating card-company merchant review saved 20 FTE.
※ Reconstruction with a fictional shop — not a real merchant or review screen.
Built a browser-use review agent that browses merchant websites like a human reviewer and collects business details, terms and the payment path
Automated card-company merchant review (payment-path capture), saving 20 FTE, and applied the same agent to Toss Pay merchant review (an 8-person workload)
Had AI run reviewers' browser checks the same way they did, targeting same-day results for a manual review that averaged about 3 days
Reviewers opened each merchant website by hand to check and capture business details, terms and the payment path; a manual review took about three days on average. The automation targets same-day results.
Every merchant site is structured differently, so reviewers had to hunt for the right pages one by one.
02Key Challenges
C1 Finding every piece of review evidence on sites that are all structured differently
C2 Not burdening or endangering third-party merchant servers
C3 Deciding when enough has been collected and stopping
03Contributions
Browser-use review agentC1C3
Designed a browsing loop (observe → reason → act) that reads the screenshot and page text together and decides where to look next
Set collection goals per review item (business details, terms, payment path) so the agent picks its next action from what is still missing
Made the agent stop on its own once it had what it needed, keeping captured screens and text as review evidence
Safety for third-party serversC2
Read public pages only, with one browser per merchant going one page at a time, so merchant servers see no more load than a single visitor
04Tech Stack
Framework / Platform
browser-use, Headless Chromium, Multimodal LLM
Methodology
Agentic browsing, Prompt engineering
05Results
Saved 20 FTE by automating card-company merchant review (payment-path capture)
Applied the same agent to Toss Pay merchant review (an 8-person workload)
T-02Viva Republica (Toss)Jun 2026 – Present
Inspection Automation Platform
A platform where AI automates the steps between an inspection request and a working, improving inspection agent. AI handles everything from organizing requirements to refining prompts and re-validating, and shows users where the data conflicts with the current direction, so they only set the direction and approve.
RolePlatform design and development
10+departmentsused as a shared module across inspection tasks
01 / 06
AI organizes the requirements
AI gathers material scattered across chat threads, wiki pages, sheets and owner notes, drafts the inspection requirements, and turns gaps into questions.
02 / 06
AI drafts the agent
From the requirements, AI proposes the components and their order and builds a draft inspection agent, flagging anything it cannot build or that is missing.
03 / 06
Verdicts, evidence, versions
Each run keeps the verdict, evidence, run history and version together. Reviewers can check results right away, and these records become the data for the next improvement.
04 / 06
Compare with review data
Set against real reviewers’ decisions, the results show where the two disagree. This is where the improvement cycle starts.
05 / 06
AI fixes, people set the direction
For gaps a prompt can fix, AI drafts a change with the agent’s structure in mind and re-runs the same data to compare. Where the data conflicts with the current criteria, it shows the user the cases and asks which way to go.
06 / 06
A new version, then the next cycle
Approved changes become a new version, and what is left becomes the next cycle’s question. 10+ departments have applied the platform to their inspection work.
GoalCheck that merchant sites show the required notices
ItemsRefund notice · business info · payment methods
QuestionDo overseas merchants follow the same rules?
Step
Component (AI proposal)
Uses
1
Collect notices from merchant pages
Input format
2
Judge refund, business info, payment
Policy 3.2 · 2.1 · 4.1
3
Write up verdicts and evidence
Result format
Missing: rules for overseas merchants → sent back as a question
Merchant AViolation
BasisPolicy 3.2 — refund policy required
ObservedNo refund notice on product or terms pages
Historyv1 · run log saved
Merchant COK
ObservedName and registration number in the footer
Case (example)
AI
Reviewer
Fixable by the prompt · 1
AI drafts a prompt change: payment methods shown only in an image go to a person instead of being judged
+ Notice only in an image → needs reviewRe-run on the same data → 1 resolved
Direction check needed · 2
Current rule: refund policy must be on the product page
Data: reviewers accept a refund clause that is only in the terms (2 cases)
This is about the rule, not the prompt. Which way should it go?
Keep the ruleUpdate the rule to match
v14/8agree with reviewers
→
v27/8same example data
Changes: 1 prompt fix · 1 rule update (approved by the user)
1 difference left (overseas payment) → carried into the next cycle as a question
10+departments apply it to their inspection work
01 · AI organizes the requirements — AI gathers material scattered across chat threads, wiki pages, sheets and owner notes, drafts the inspection requirements, and turns gaps into questions.
02 · AI drafts the agent — From the requirements, AI proposes the components and their order and builds a draft inspection agent, flagging anything it cannot build or that is missing.
03 · Verdicts, evidence, versions — Each run keeps the verdict, evidence, run history and version together. Reviewers can check results right away, and these records become the data for the next improvement.
04 · Compare with review data — Set against real reviewers’ decisions, the results show where the two disagree. This is where the improvement cycle starts.
05 · AI fixes, people set the direction — For gaps a prompt can fix, AI drafts a change with the agent’s structure in mind and re-runs the same data to compare. Where the data conflicts with the current criteria, it shows the user the cases and asks which way to go.
06 · A new version, then the next cycle — Approved changes become a new version, and what is left becomes the next cycle’s question. 10+ departments have applied the platform to their inspection work.
※ Reconstruction with a fictional inspection task; merchants, rules and numbers are illustrative, not real screens or data.
Built an inspection automation platform where AI refines prompts from data and the agent's structure and shows users where the data conflicts with the current direction, making the improvement cycle easy
Automated the in-between steps of building an inspection agent with AI features (requirement drafting, agent drafts, versioned experiments); used as a shared inspection module by 10+ departments
10+departmentsused as a shared module across inspection tasks
01Background & Goals
Every inspection task needed its own agent, and when results were off, people rewrote the prompts by hand and ran them again.
When results were off, it was hard to tell whether the prompt was wrong or the policy and criteria (the direction) themselves needed to change.
Requirements, policies and past cases were scattered across chat threads, documents and sheets, so even preparing to build an agent took a long time.
02Key Challenges
C1 Telling prompt problems apart from problems where the policy (the direction) must change
C2 Cutting repeated manual steps such as organizing requirements, assembling agents and editing prompts
C3 Managing each task's policies, data and formats together and improving them by version
03Contributions
AI-driven improvement cycleC1
AI finds recurring format and evidence errors in run logs and review data, drafts prompt changes that fit the agent's structure, and re-runs the same data to compare
Errors a prompt cannot fix are shown to users as places where the data conflicts with the current direction (policy) or the policy has gaps, with supporting cases, affected scope and a proposed fix
Users set the direction and approve; AI does the editing, re-running and comparing, so the cycle is easy to repeat
AI features that cut the steps in betweenC2
AI gathers scattered material (chat threads, documents, sheets, owner notes), drafts the inspection requirements, and turns gaps into questions
From the requirements, AI proposes the components and their order and builds a draft inspection agent, flagging what it cannot build and what is missing
AI generates each task's prompts and supports versioned experiments, so prompt versions can be compared on results
Platform architecture and shared moduleC2C3
Ran inspection agents on LangGraph; built the intermediate AI features (requirements, agent drafts, prompt learning, policy refinement) as Claude Agent SDK services
Kept policies, data, cases and input/output formats together as reference information, and connected each task's agent in a develop → run → improve flow
Stored each run's verdict, evidence, run history and version, and fed reviewers' final decisions (confirm, edit, hold) into the next improvement
Used as a shared inspection module across inspection tasks in 10+ departments
T-03Viva Republica (Toss)Mar 2026 – Jun 2026
Ad Review Agent
Built an ad review agent that generates a review guideline from a one-line ad policy and has AI refine it from real review data.
RoleDesign and development
+25 ppaverage accuracy gainad-copy typo policy pilot · mean gain of the refinement and held-out sets (36 ads)
01 / 06
A one-line policy
Ad policies are a line or two — "restrict expressions that look like typos." That gives an agent no criteria, boundaries or examples, and leaves out the judgment only reviewers know.
02 / 06
Generate the guideline
Grounded in the policy source, the one-line policy becomes a guideline that says what to reject and what to accept, with boundaries and references.
03 / 06
Compare with reviewers
The refinement ads are judged by the guideline and compared with reviewers’ verdicts. Most disagreements were false rejections of acceptable ads; this first version is the baseline.
04 / 06
Revision 1 from the gap
AI finds the common pattern behind the false rejections (compounds and amount units that are normal in ads) and revises the guideline. Accuracy rose, but loosening it too far caused a miss.
05 / 06
Revision 2 tightens the edge
Ad-style phrasing is marked OK, and compounds of unrelated words that hurt readability are rejected again. People only checked the revisions.
06 / 06
Unseen ads improve too
Accuracy also rose on held-out ads never used for refinement; across the refinement and held-out ads it rose 25 pp on average. The gain carries over to unseen ads, and this approach became the Inspection Automation Platform’s improvement cycle.
Reconstructed screen · synthetic data
guideline.refine(policy, reviews)
Review policy adminv1
Policy source (summary): ads that needlessly use misspelled expressions may be restricted.
Generate guideline
Find the basis in the policy source
Write rejection criteria and examples
Draft the violation / review / OK boundary
2.1 Violation types
2.2 Rejection criteria · examples
Typo — Discuont → DiscountUnfinished — Sign up tod → Sign up todaySpacing — reject any rule violation
2.3 Policy basis
2.4 Decision guide
2.5 Decision context
2.6 Violation / review / OK boundary
Ad copy (synthetic)
Doc
Reviewer
Re-checked on ads never used for refinement
18 refinement ads+17 pp
18 held-out ads+33 pp
Average+25 pp
Unseen ads improved too → the gain carries over to new ads · single-policy pilot
01 · A one-line policy — Ad policies are a line or two — "restrict expressions that look like typos." That gives an agent no criteria, boundaries or examples, and leaves out the judgment only reviewers know.
02 · Generate the guideline — Grounded in the policy source, the one-line policy becomes a guideline that says what to reject and what to accept, with boundaries and references.
03 · Compare with reviewers — The refinement ads are judged by the guideline and compared with reviewers’ verdicts. Most disagreements were false rejections of acceptable ads; this first version is the baseline.
04 · Revision 1 from the gap — AI finds the common pattern behind the false rejections (compounds and amount units that are normal in ads) and revises the guideline. Accuracy rose, but loosening it too far caused a miss.
05 · Revision 2 tightens the edge — Ad-style phrasing is marked OK, and compounds of unrelated words that hurt readability are rejected again. People only checked the revisions.
06 · Unseen ads improve too — Accuracy also rose on held-out ads never used for refinement; across the refinement and held-out ads it rose 25 pp on average. The gain carries over to unseen ads, and this approach became the Inspection Automation Platform’s improvement cycle.
※ Ad copy and policy text are synthetic examples of patterns seen in the pilot. Numbers come from a single-policy pilot (36 real ads).
Built an ad review agent that turns a short ad policy into a review guideline with criteria, boundaries and examples
Added AI refinement that finds where the guideline disagrees with reviewers and revises it, raising average accuracy by 25 pp on the pilot policy
Accuracy also rose on held-out ads never used for refinement, so the gain carries over to unseen ads
+25 ppaverage accuracy gainad-copy typo policy pilot · mean gain of the refinement and held-out sets (36 ads)
01Background & Goals
Ad policies are often a line or two, but an AI reviewer needs a guideline with criteria, boundaries and examples, and people had to write one for every policy.
When the guideline was stricter or looser than real reviewers, it rejected good ads or missed violations, and finding where it went wrong was hard.
02Key Challenges
C1 Turning a short policy into workable review criteria
C2 Closing the gap between the guideline and reviewers with data
C3 Avoiding overfitting to the refinement data
03Contributions
Automatic guideline generationC1
Found the basis in the policy source text and turned a short policy into a guideline with decision criteria, violation/OK boundaries and policy references
Data-driven refinementC2C3
Had AI compare the guideline's decisions with reviewers', find the patterns behind the disagreements and revise the guideline (e.g. marking compounds, amount units and ad-style phrasing common in ads as OK)
Split refinement and held-out data to confirm the improvement holds on unseen ads
People only checked the revisions; nobody edited the guideline by hand
04Tech Stack
Framework / Platform
LLM, Prompt engineering
Methodology
Data-driven prompt refinement, Holdout validation
05Results
Raised average accuracy by 25 pp on an ad-copy typo policy pilot (36 ads)
Accuracy also rose on held-out ads never used for refinement, so the gain holds on new ads
This approach became the prompt-refinement cycle of the Inspection Automation Platform
02 · SK Telecom
Conversational agents behind A-dot
For A-dot and T Membership, I designed and built a scheduling agent that handles several requests in one sentence, a movie agent that finishes the booking inside the conversation, and a main agent that offers 17 functions through chat.
SK TelecomAI Agent Department · Manager · Apr 2022 – Oct 2025
67%
registrations via agent
66K of 98K schedule registrations
60→80%
function-call success
across 17 functions, after fine-tuning and prompt work
12%
booking conversion
T Membership movie booking · bookings / visitors
S-01SK TelecomApr 2025 – Oct 2025
A-dot Scheduling Agent
Designed and built A-dot's scheduling agent, which lets users manage events and reminders through conversation and splits multi-intent requests into sub-tasks to handle them quickly and accurately.
RoleAgent design and development
2.5×schedule MAU growth203K → 511K (Jan → Sep 2025)
67%registrations via agent66K of 98K schedule registrations
01 / 06
Three intents in one sentence
“Add a dentist appointment at 3 tomorrow, push Friday’s meeting back, and tell me this week’s schedule.” Create, update and read are mixed in one sentence. First the intents are split and details like time and target are extracted.
02 / 06
Plan: sub-tasks and order
The request is split into ordered sub-tasks. The meeting must be found before it is moved, so “find → update” runs in sequence; the rest are independent.
03 / 06
Execute: independent tasks in parallel
Tasks that do not affect each other run at the same time; only ordered tasks run in sequence. Latency is set by the longest lane, not the sum of every task.
04 / 06
RAG for finding events
Candidates are filtered by owner, subscription and date, then split into N chunks searched by parallel LLM calls. Each hit is verified with evidence, which rejects look-alikes such as a team dinner.
05 / 06
Refine: re-plan from results
Execution results feed a re-planning step. The meeting that was found (Fri 10:00) fixes the update task’s arguments, and the remaining work is organized and distributed again.
06 / 06
One answer, registered accurately
All three results come back in one answer. Recurring events and reminders use the iCalendar and ISO 8601 standards for accuracy, and 67% of schedule registrations went through this agent.
Reconstructed screen · synthetic data
agent.plan_and_execute(utterance)
Schedule assistantChat
Wed, Sep 24
Add a dentist appointment tomorrow at 3 PM,push Friday’s team meeting back 30 minutes,and tell me what’s on this week
✓Analyzing intents3 intents found
✓PlanningPlanned 4 sub-tasks
✓Running 3 lanes at onceRan 3 lanes in parallel
✓Finding “Friday team meeting”Found Fri 10:00 meeting
✓Re-planning from resultsRe-planned
✓Handled in 5 phases
All three are done.
✓Dentist added · Thu 15:00reminder 30 min before
✓Team meeting 10:00 → 10:30
✓4 events left this week
Thu 15:00Dentistnew
Fri 10:30Team meetingmoved
Fri 19:00Team dinner
Sat 11:00Family lunch
Intent
Plan
Execute
RAG
Refine
Answer
intent_analysis()1 request → 3 intents
1
CreatecreateDentist appointment
when: 09-25 15:00
2
UpdateupdateFriday team meeting
shift: +PT30M
3
ReadlistThis week
range: 09-22 ~ 09-28
Each intent needs different tools and data → plan sub-tasks
plan()4 sub-tasks · 2 stages
Stage 1 · togetherStage 2
1create_eventno depsDentist · tomorrow 15:00
2afind_event“Friday team meeting”
3list_eventsno depsThis week
needs
2bupdate_event+30 min · the meeting 2a finds
Independent tasks share stage 1; only tasks that need a result wait for the next stage
execute()Example
0s1s2s3s4s
Parallel
1create
create
2find→update
findupdate
3list
list
If run one after another
Sequential
1.7sparallel · longest lane
3.7ssequential · sum of all
find_event()Example
1Filter
312eventsOwner: meNo subscriptionsSep 22–2824candidates
2Parallel search24 → 4 chunks · 4 LLM calls at once
#1LLM0
#2LLM1
#3LLM0
#4LLM1
3Verify + evidence
Fri 10:00 · Weekly team meetingEvidence: “team meeting” in title · Friday · own calendar✓
Fri 19:00 · Team dinnerEvidence: a dinner, not a meeting✕
refine()results → re-plan
2bupdate_eventarguments resolved
TargetunknownWeekly team meeting · Fri 10:00
Time10:0010:30
iCalendar · RFC 5545ISO 8601 duration
1create_eventBEGIN:VEVENTSUMMARY:DentistDTSTART:20250925T150000tomorrow 15:00BEGIN:VALARMACTION:DISPLAYTRIGGER:-PT30M30 min beforeEND:VALARMEND:VEVENT
2update_eventthis occurrence of a seriesRRULE:FREQ=WEEKLY;BYDAY=FRseries · every FriRECURRENCE-ID:20250926T100000this oneDTSTART:20250926T10300010:00 → 10:30
01 · Three intents in one sentence — “Add a dentist appointment at 3 tomorrow, push Friday’s meeting back, and tell me this week’s schedule.” Create, update and read are mixed in one sentence. First the intents are split and details like time and target are extracted.
02 · Plan: sub-tasks and order — The request is split into ordered sub-tasks. The meeting must be found before it is moved, so “find → update” runs in sequence; the rest are independent.
03 · Execute: independent tasks in parallel — Tasks that do not affect each other run at the same time; only ordered tasks run in sequence. Latency is set by the longest lane, not the sum of every task.
04 · RAG for finding events — Candidates are filtered by owner, subscription and date, then split into N chunks searched by parallel LLM calls. Each hit is verified with evidence, which rejects look-alikes such as a team dinner.
05 · Refine: re-plan from results — Execution results feed a re-planning step. The meeting that was found (Fri 10:00) fixes the update task’s arguments, and the remaining work is organized and distributed again.
06 · One answer, registered accurately — All three results come back in one answer. Recurring events and reminders use the iCalendar and ISO 8601 standards for accuracy, and 67% of schedule registrations went through this agent.
※ Reconstruction with a fictional calendar and numbers — not the actual service screen.
Designed and built an agent that handles multi-intent scheduling requests by splitting them into sub-tasks and re-planning from the results (Plan-and-Execute + Refine)
Kept multi-intent requests fast by running independent sub-tasks and schedule searches in parallel
67% of schedule registrations came through the agent (66K of 98K)
2.5×schedule MAU growth203K → 511K (Jan → Sep 2025)
67%registrations via agent66K of 98K schedule registrations
01Background & Goals
We built an agent that handles A-dot's events and reminders through conversation.
The goal was to handle multi-intent requests such as "add a dentist appointment at 3 tomorrow and push Friday's meeting back 30 minutes" quickly and accurately.
02Key Challenges
C1 Multi-intent requests and response latency
C2 Integration with external calendars (Outlook, Google Calendar) and subscription calendars
C3 Accurate handling of time information such as recurring events and reminders
03Contributions
Agent architecture with the Plan-and-Execute pattern and a Refine stepC1
Plan: splits the request into sub-tasks and orders them
Execute (sequential): completes dependent sub-tasks in order
Execute (parallel): runs independent sub-tasks in parallel to cut latency
Refine: re-plans from the execution results to raise task accuracy
External and subscription calendar integrationC2
Designed the flow so the LLM distinguishes external calendars (Outlook, Google Calendar) from subscription calendars (benefit and event schedules) and decides whether each is linked and can be read, changed or deleted
RAG for schedule retrievalC1
Filtering: narrows candidate events by owner, subscription and date range
Parallel search: splits candidates into N chunks and calls the LLM in parallel to find the requested events within the latency budget
Verification: checks that each extracted event matches the request and extracts the supporting evidence to improve accuracy
Prompt engineering for time informationC3
Designed prompts that use the iCalendar standard and ISO 8601 durations so recurring events and reminders are handled precisely
04Tech Stack
Framework / Platform
LangGraph, Gradio, FastAPI
Methodology
Prompt engineering, RAG, Plan-and-Execute
05Results
67% of schedule registrations came through the agent (66K of 98K)
Service-wide schedule MAU grew 2.5× (203K in Jan → 511K in Sep 2025)
Designed A-dot's flagship LLM agent, which offers 17 functions such as exchange rates, weather and directions through conversation, and raised its function-call accuracy through fine-tuning and prompt design.
60→80%function-call successacross 17 functions, after fine-tuning and prompt work
01 / 06
An everyday question
“Is Euljiro 3-ga Station crowded right now?” The agent first decides whether to just chat or to call one of its 17 functions.
02 / 06
Call a function, don’t make it up
The LLM picks search_congestion and fills the location argument with “Euljiro 3-ga”. Colors in the log on the right mark the message role: user · function_call · function · assistant.
03 / 06
Function results as Markdown
The system returns the function result formatted as Markdown. Unlike code-laden JSON, Markdown is easy for the model to read and quote, so answers stay faithful to the result. Use the toggle to compare.
04 / 06
An answer plus a UI card
The agent explains the result in words, and the screen shows the same result as a congestion card.
05 / 06
Multi-turn: where is “there”?
In “fastest way from there to Guro”, “there” is Euljiro 3-ga from the previous turn. The agent fills the argument from context, calls search_route and answers with a route card.
06 / 06
17 functions, success 60% → 80%
It covers 17 functions such as exchange rates, weather, time, directions, news and subway congestion. Designing multi-turn function-calling training data, fine-tuning GPT models and refining prompts raised function-call success from 60% to 80%.
## Route · Euljiro 3-ga → Guro
- Line 2 → change at Sindorim → Line 1
- About 27 min
assistant
Take Line 2 and change to Line 1 at Sindorim — about 27 min.
functions[17]Chat + 17 functions
Exchange rateget_exchange_rate
Weatherget_weather
Timeget_time
Newssearch_news
Subway congestionsearch_congestion
Directionssearch_route
+11more functionscalled in this chat
Function-call successacross 17 functions
Before
60%
After
80%
Fine-tuned GPT-3.5 Turbo · GPT-4o mini + prompt engineering
Markdown results
Merged functions
Sequential calls
NER for time
Multi-turn data
01 · An everyday question — “Is Euljiro 3-ga Station crowded right now?” The agent first decides whether to just chat or to call one of its 17 functions.
02 · Call a function, don’t make it up — The LLM picks search_congestion and fills the location argument with “Euljiro 3-ga”. Colors in the log on the right mark the message role: user · function_call · function · assistant.
03 · Function results as Markdown — The system returns the function result formatted as Markdown. Unlike code-laden JSON, Markdown is easy for the model to read and quote, so answers stay faithful to the result. Use the toggle to compare.
04 · An answer plus a UI card — The agent explains the result in words, and the screen shows the same result as a congestion card.
05 · Multi-turn: where is “there”? — In “fastest way from there to Guro”, “there” is Euljiro 3-ga from the previous turn. The agent fills the argument from context, calls search_route and answers with a route card.
06 · 17 functions, success 60% → 80% — It covers 17 functions such as exchange rates, weather, time, directions, news and subway congestion. Designing multi-turn function-calling training data, fine-tuning GPT models and refining prompts raised function-call success from 60% to 80%.
※ Reconstruction with a fictional conversation and numbers — not the actual service screen. Only the function-call success rate (60% → 80%) is a real result.
Designed the training-data schema (multi-turn, function calling) and fine-tuned and evaluated GPT models to raise function-call accuracy and multi-turn quality
Split hallucination-prone or low argument-accuracy functions into sequential calls and extracted time arguments with NER first, reducing time-parsing errors
Raised function-call success across 17 functions from 60% to 80% through data design, fine-tuning and prompt work
Role · Agent design, prompt engineering, training-data design, fine-tuning
60→80%function-call successacross 17 functions, after fine-tuning and prompt work
01Background & Goals
A-dot's flagship LLM agent, offering everyday conversation plus 17 functions such as exchange rates, weather, time, directions, news and subway congestion.
02Key Challenges
C1 Function-call accuracy and hallucination
C2 API cost and latency
C3 Multi-turn conversation quality
03Contributions
Agent design for accuracy and UI/UX integrationC1
Consolidated similar functions to simplify function selection
Let the conversation use in-app context such as the current playlist and UI screen
Split functions with low argument accuracy into sequential calls
Prompt engineering for cost and answer accuracyC1C2
Kept prompts minimal to cut API cost and used fine-tuning to instill answer style
Switched argument extraction for hallucination-prone time functions to NER
Fine-tuning and evaluationC3
Defined the training-data format and generated multi-turn dialogue data
Fine-tuned GPT models on Azure OpenAI
Ran quantitative evaluation (function selection and accuracy, argument selection and extraction) and qualitative evaluation (multi-turn fluency, accuracy of function-result answers, handling of sensitive and inappropriate requests)
04Tech Stack
Framework / Platform
Azure OpenAI
Methodology
Prompt engineering, Function calling, Fine-tuning, NER
05Results
Raised function-call success across 17 functions from 60% to 80% through fine-tuning and prompt engineering
Reduced hallucination and improved time-parsing accuracy by extracting time arguments with NER and analyzing them sequentially
Designed an agent that recommends, searches and books movies as naturally as talking to cinema staff, built as an LLM workflow where chat and UI work together.
RoleUX & LLM workflow design, RAG development
12%booking conversionT Membership movie booking · bookings / visitors
110Kcumulative usersmovie-booking agent
8,000MAUT Membership movie booking
01 / 05
Like talking to cinema staff
The user just says what they want in the membership app’s chat. The message lands on the LLM message stack, and the agent starts collecting what a booking needs (slots).
02 / 05
Slot filling with personalization
A fine-tuned model uses function calling to pull date, time and area into slots. The user’s location and recently visited cinemas suggest a cinema first, and a state-management module keeps slots intact across long conversations.
03 / 05
RAG for movie search
A Trie built from a title-synonym dictionary tags movie titles, the LLM extracts keywords and metadata, and vector search with metadata filters finds candidates. Finally the LLM picks the movies that fit.
04 / 05
UI taps join the same workflow
When the user taps a showtime instead of typing, the result becomes a message in the LLM prompt. The next reply knows what was just tapped, so chat and UI flow as one. Try tapping a time yourself.
05 / 05
All slots filled, ready to book
Once party size and seats are filled, a booking card appears. This flow reached 12% booking conversion (completed bookings / visitors).
Reconstructed screen · synthetic data
movie_agent.run(chat, ui_events)
ChatUI tapProfile
9:41
‹Membership · MoviesAI
Hi! What would you like to see?
I want to see a movie in Euljirotomorrow evening
I’ll check Euljiro Cinema, where you went recently. Any movie in mind?
Is starvoyage still on? Something calm is fine too
Tomorrow evening at Euljiro Cinema:
Star VoyageSci-fi · 124 minSummer ShowerDrama · 108 minThe Twelfth HaltDrama · 116 min
Star Voyage · tomorrow
Star Voyage at 19:40, got it. How many of you?
Two of us, seats in the middle
Star VoyageTomorrow 19:40 · Euljiro Cinema Hall 32 people · F7, F8
01 · Like talking to cinema staff — The user just says what they want in the membership app’s chat. The message lands on the LLM message stack, and the agent starts collecting what a booking needs (slots).
02 · Slot filling with personalization — A fine-tuned model uses function calling to pull date, time and area into slots. The user’s location and recently visited cinemas suggest a cinema first, and a state-management module keeps slots intact across long conversations.
03 · RAG for movie search — A Trie built from a title-synonym dictionary tags movie titles, the LLM extracts keywords and metadata, and vector search with metadata filters finds candidates. Finally the LLM picks the movies that fit.
04 · UI taps join the same workflow — When the user taps a showtime instead of typing, the result becomes a message in the LLM prompt. The next reply knows what was just tapped, so chat and UI flow as one. Try tapping a time yourself.
05 · All slots filled, ready to book — Once party size and seats are filled, a booking card appears. This flow reached 12% booking conversion (completed bookings / visitors).
※ Reconstruction with a fictional app, cinemas and movies — not the real service screen.
Designed a UI-integrated agent that lets users move between chat and buttons to find and book movies, feeding the results of UI actions into the LLM workflow
Built RAG that finds the movie a user asks for in conversation, and automated metadata and keyword extraction from daily movie updates to keep search current
Fine-tuned GPT models on data collected from service logs to raise function-calling accuracy and multi-turn fluency
12% booking conversion on T Membership (completed bookings / visitors); 110K cumulative users of the movie-booking agent
12%booking conversionT Membership movie booking · bookings / visitors
110Kcumulative usersmovie-booking agent
8,000MAUT Membership movie booking
01Background & Goals
A conversational agent for T Membership movie booking, meant to make getting recommendations, searching and booking as easy as talking to cinema staff.
The service combines chat with UI elements such as buttons, so the UI/UX and the LLM workflow had to flow into each other naturally.
02Key Challenges
C1 Slot filling through conversation, with personalization
C2 Integrating the UI/UX with the LLM workflow
C3 Movie search accuracy, latency and API cost
03Contributions
Slot-filling agentC1
Designed a function-calling LLM workflow that informs the user while collecting what the booking needs
Personalized recommendations using the user's location, recently visited cinemas and favorite theaters
Added a state-management module so slots collected in long conversations are not lost
RAG for movie searchC3
Preprocessing: tags movie titles in the request with a Trie built from a title-synonym dictionary
Keyword extraction: uses the LLM to extract movie metadata and keywords from the request
Retrieval: vector search with metadata filtering on the extracted keywords
Final pick: uses the LLM to choose the movies that fit the request
Automated movie-metadata pipelineC3
Extracted keywords, summaries and metadata from daily movie updates with Airflow, an LLM and preprocessing modules
Converted the results to vectors and synced them to the vector DB
UI/UX and LLM workflow integrationC2
Fed the results of UI actions (buttons, etc.) into the LLM workflow's prompt so both flow into each other naturally
Log-based data collection and trainingC3
Collected training data from service logs and fine-tuned GPT models on Azure OpenAI to raise function-calling accuracy and multi-turn fluency
04Tech Stack
Framework / Platform
Azure OpenAI, Datadog, Airflow
Methodology
Prompt engineering, Function calling, RAG
05Results
8,000 MAU for T Membership movie booking; 110K cumulative users of the movie-booking agent
Cupid: Answer Recommendation for Similar Questions
Built the retrieval and recommendation pipeline and BERT-based models that surface existing answers to similar questions in a location-based Q&A community.
RoleModel development, recommendation and evaluation pipelines
16→31%AI auto-answer rateduplicate questions, Aug 1 → Aug 21, 2022
38→57%positive feedback"helpful" ratio on AI answers
Reconstructed screen · synthetic data
cupid.find_answers(question)
01Question
Neighborhood Q&AYeoksam
Neighbor · just now
Any quiet cafes near Gangnam Station?
Somewhere to work on a laptop for a few hours.
↻AI is looking for answers to similar questions
AI answer
2nd-floor cafe behind Exit 10, calm on weekday afternoons.
3 answers from similar questions
02Preprocess
NER
Any quiet cafes near Gangnam Station[LOC]?
TopicPlaces›Cafe
FilterNear Gangnam Station
Embed
03RetrieveExample
All questions1,000
Location·topic120
Top hits20
BM25Cosine
Quiet cafe open today?0.86
Cafe with outlets?0.79
Quiet cafe to study?0.77
Cafe for a quiet chat?0.71
04Re-rank
1
Is the quiet cafe at Gangnam open today?AI answerA sign said it’s closed today.
Dropped
2
Gangnam cafe with lots of outlets?AI answerSide-street cafes are calmer and have outlets.
Dropped
3
Quiet cafe to study near Gangnam?AI answer2nd-floor cafe behind Exit 10, calm on weekday afternoons.
Dropped
4
Cafe for a quiet chat near Gangnam?AI answerThe book cafe by the underground mall has roomy seats.
Dropped
Re-sorted by containment and Q–A consistency
AI auto-answer rateduplicate questions · Aug 1→21, 2022
16%→31%
01 · A question in the neighborhood Q&A — A question is posted to the location-based community. If similar questions already have answers, the AI surfaces them first.
02 · Preprocess: extract, classify, mask, embed — BERT-based NER and hierarchical classifiers pull out the location and topic for filtering; place names are masked as [LOC] and a Siamese-trained embedding model turns the question into a vector.
03 · BM25 + cosine retrieval — Location and topic narrow the candidates, then Elasticsearch combines BM25 and cosine similarity to find similar questions.
04 · Re-rank, then answer — Question containment and question–answer consistency re-rank the hits, dropping look-alikes, and the top answers are shown. The AI auto-answer rate for duplicate questions rose from 16% to 31%.
※ Reconstruction with a fictional neighborhood Q&A app; scores and candidate counts are illustrative.
01 / 04
A question in the neighborhood Q&A
A question is posted to the location-based community. If similar questions already have answers, the AI surfaces them first.
02 / 04
Preprocess: extract, classify, mask, embed
BERT-based NER and hierarchical classifiers pull out the location and topic for filtering; place names are masked as [LOC] and a Siamese-trained embedding model turns the question into a vector.
03 / 04
BM25 + cosine retrieval
Location and topic narrow the candidates, then Elasticsearch combines BM25 and cosine similarity to find similar questions.
04 / 04
Re-rank, then answer
Question containment and question–answer consistency re-rank the hits, dropping look-alikes, and the top answers are shown. The AI auto-answer rate for duplicate questions rose from 16% to 31%.
16→31%AI auto-answer rateduplicate questions, Aug 1 → Aug 21, 2022
38→57%positive feedback"helpful" ratio on AI answers
Built a pipeline that recommends answers already given to similar questions (location/topic filtering, BM25 + embedding search, answer re-ranking)
Fine-tuned BERT for the pipeline's embedding, hierarchical topic-classification and place-extraction (NER) models
Researched speech recognition and NLP at Korea University’s DSBA lab (Interspeech 2022), then improved ASR decoding at Hyundai.
Interspeech 2022 · First author·Sep 2022
K-Wav2vec 2.0: Korean ASR with Grapheme–Syllable Joint Decoding
Proposed a Korean-adapted Wav2vec 2.0 with joint decoding over graphemes and syllables and further pre-training on Korean audio.
Reconstructed screen · synthetic data
kwav2vec.decode(wav)
Speech “안녕하세요”16 kHz
Wav2vec2.0GraphemeSyllableJoint
Head output · waitingExample
Beam candidates
안녕하세요✓
안녕하새요✓
안녕하세여✓
안영하세요✓
01 · Speech → Wav2vec 2.0 — A Wav2vec 2.0 encoder turns Korean speech into frame-level representations. The English-pretrained model was further pre-trained on unlabeled Korean speech.
02 · Grapheme head: ㅇㅏㄴㄴㅕㅇ… — The first head emits graphemes (jamo). Its small units can spell unseen words, but it can confuse letters that sound alike, such as ㅐ and ㅔ.
03 · Syllable head: 안·녕·하·세·요 — A second head on the same encoder emits syllables (multi-task). It captures word shapes well, yet can still pick a wrong syllable, as in this example.
04 · Joint decoding: plausible in both — Beam-search scores from both heads are added to re-rank the candidates. A candidate strong in only one head drops, and ‘안녕하세요’ (“hello”), plausible in both, wins.
※ Synthetic example; candidates and scores are illustrative.
Designed a multi-task head that learns graphemes and syllables together and a joint decoder that combines them, reflecting how Korean syllables are built from graphemes
Further pre-trained an English-pretrained model on unlabeled Korean speech to improve low-resource performance
Korea Univ. DSBA × Hanwha Systems (ICT) industry–academia project·Jan 2020 – Sep 2021
Predictive Maintenance for Chemical Processes
Built a predictive-maintenance system that detects early warning signs in chemical-plant equipment and explains them, so engineers can act before a failure.
Reconstructed screen · synthetic data
plant.monitor(pump_07)
5 sensors · liveExampleNormal
Lead time
Vibrationmm/s
SounddB
Speedrpm
Temp.°C
Pressurebar
Recon. errorRNN-AEThreshold 0.50Alarm
Pre-failureFault
LSH matchesLikely cause cavitation
Cavitationsim 0.93
Cavitationsim 0.88
Normalsim 0.41
01 · Five sensors, streaming — Vibration, sound, rotation speed, temperature and pressure readings stream in. An RNN auto-encoder reconstructs the normal pattern, and the reconstruction error stays below the threshold.
02 · Inject a fault: cavitation (example) — Suppose cavitation starts in a pump. Vibration and sound slowly drift out of their normal range — still hard to spot by eye.
03 · Alarm when the error crosses the line — The less the pattern can be reconstructed as normal, the larger the error; the alarm fires when it crosses the threshold, giving engineers time to act before a failure.
04 · Explain why it is anomalous — LSH finds past windows similar to the current one and compares them with known anomalous and normal cases to show a likely cause.
※ Reconstruction with a fictional unit and synthetic sensor data; all values are illustrative.
Detected warning signs early with an RNN auto-encoder anomaly detector and RNN time-series forecasting, so engineers could act before a failure
Built an analysis system that finds similar past patterns with LSH and separates normal from anomalous patterns
Korea Univ. DSBA × Hanwha Systems (ICT) industry–academia project · Jan 2020 – Sep 2021
Predictive Maintenance for Chemical Processes
Role · Anomaly-pattern analysis, anomaly-detection algorithms
01Background & Goals
An industry–academia project to detect warning signs in equipment, find their causes and let engineers act before a failure.
We applied anomaly detection, time-series and pattern analysis to chemical-plant sensor data (vibration, sound, rotation speed, temperature, pressure).
02Contributions
Detection and forecasting models
Built an RNN auto-encoder anomaly detector to keep equipment running
Forecast future patterns from historical data with an RNN time-series model
Explanation and monitoring
Found similar past patterns with LSH and separated normal from anomalous patterns for analysts
Built a real-time monitoring system driven by sensor status updates
Korea University (graduate project)·Apr 2020 – Jul 2020
Open-Domain Korean Question Answering
Built a Korean open-domain QA system that gathers candidate documents from search engines and finds the answer span with BERT-based reading comprehension.
Retrieve candidate paragraphs, then read them to extract the answer span · Flask web demo
Built a retrieval pipeline that collects candidate documents from search engines and splits them into paragraphs
Evaluated BERT, KoBERT and HanBERT span-extraction models quantitatively and qualitatively to choose the final model
Studied how combining back-translation augmentation with task-adaptive pre-training improves the accuracy and robustness of text classifiers trained on little data.
Task-adaptive pre-training on back-translated paraphrases, then fine-tuning · more accurate, robust classifiers from little data
Further pre-trained (TAPT) on task data augmented with back-translation, to improve text-classification accuracy and robustness when data is scarce
Implemented back-translation augmentation (nucleus sampling for diverse paraphrases) and compared it with EDA, TF-IDF and embedding-swapping augmentation
SK Inc. C&C (now SK AX) · Jan 2015 – Mar 2020. Built KDB’s e-finance services (FIDO fingerprint authentication, remote ID verification, Smart OTP), then corporate-distress early warning, robo-advisor and news-based market-prediction models.
SK C&C·Jul 2019 – Mar 2020
Market Caster: News-based Market Prediction
Built the event-extraction, prediction and sentiment models for Market Caster, a news-based market-forecasting platform.
Ensembles news-event embeddings with price series to forecast indices, bonds and other markets
Predicted market moves with an ensemble model combining news-derived events and price data
Extracted and embedded (actor, action, object) events from news and predicted market events with a hierarchical event- and time-level attention model
A big-data pilot by Korea Development Bank (KDB) to recognize signs of corporate distress early.
It provides a multi-dimensionaldefault-prediction index built from stock moves, industry sentiment and an event-based distress index.
02Contributions
Event-based distress index
Extracted the core components (subject, verb, object) of news sentences about a company
Embedded their relations as vectors to assess signs of distress
Industry sentiment index
Built industry sentiment indices from news and analyzed their correlation with industry stock indices
03Tech Stack
Framework / Platform
PyTorch, scikit-learn
Methodology
NLP, Sentiment analysis, Anomaly detection
04Results
Piloted in KDB's corporate credit-risk assessment system
SK C&C (client KDB)·Jan 2016 – Mar 2017
KDB E-Finance Services
While maintaining KDB's e-finance systems, built new services prompted by regulatory change — FIDO fingerprint authentication, remote ID verification (eKYC) and Smart OTP.
Three e-finance flows launched after regulatory change · interfaces, screens and business logic
FIDO fingerprint authentication: built the app interface and signature management so customers can authenticate with a fingerprint, not only an accredited certificate
Remote ID verification (eKYC): built the browser–app flow linking ID capture, authenticity check and SMS verification, plus screens and business logic, so customers can verify identity without visiting a branch
KDB introduced new e-finance services in response to regulatory change (end of mandatory accredited certificates, permitted remote ID verification, revised e-finance supervision rules).
While maintaining KDB's e-finance systems, we built web services and app interfaces and took part in testing.
02Contributions
FIDO fingerprint authentication (Jan – Mar 2017)
Built the interface that receives the authentication result (signature) and device data from the app
Converted authentication data to banking message formats and built storage and management of signature values
Remote ID verification (eKYC) (Aug – Dec 2016)
Built the service screens and business logic with Spring MVC
Extracted and encrypted ID features with an ID-authenticity solution and built the browser–app interface for capture, feature upload and SMS verification
Smart OTP (Jan – Apr 2016)
Built the interface that collects security data and processes app-generated OTPs, plus the OTP admin menu
03Tech Stack
Framework / Platform
Spring, Android, Java, JavaScript, SQL
Methodology
FIDO, Spring MVC
04Results
FIDO fingerprint authentication, remote ID verification (eKYC) and Smart OTP launched at KDB
05 · Sharing & awards
Learning in public
Awards
Dec 20221st Prize · SK AI Challenge Document summarization for faster decision-making (fine-tuned T5)
Nov 20211st Prize · KIIE Master's Thesis Competition Multi-modal Korean emotion recognition with consistency regularization
Jan 2021Excellence Award (KPC Award) · 2nd Factory Hack Korea (MOTIE/KIAT) Forecasting oil and coolant temperature and pressure changes
Nov 2013Gold Prize · KIIE Fall Conference 2013 Optimizing routing and wavelength assignment in next-generation ROADMs
Talks · Teaching
Nov 2021NAVER CLOVA AI Tech Talk K-Wav2vec 2.0: Korean ASR based on joint decoding of graphemes and syllables
Jun 2021KIIE Spring Joint Conference 2021 Korean speech recognition with Wav2vec 2.0
Sep 2021Hyundai Mobis · Project Assistant Mentored corporate team projects on explainable AI for a plating process
Jul 2021LG Innotek · Instructor Python for machine learning and deep learning
Aug 2020LG Chem · Instructor Python for machine learning
