Agent assist reply drafting
An assistant that reads the customer's conversation and account context, retrieves relevant help centre articles and past resolved tickets, and drafts a reply or next step that the support agent edits and sends.
Supporting approachTechnique: Retrieval-Augmented Generation
While an agent works a ticket, the assistant reads the conversation and the customer’s account context, retrieves the most relevant articles and well-reviewed past resolutions, and suggests a reply or a next diagnostic step.
The agent edits the draft, checks it against the linked article, and decides whether to send it.
In the study of 5,172 agents at a Fortune 500 company selling business process software, a related assistant, fine-tuned on past conversations and linking to internal documentation, raised issues resolved per hour by 15% on average, helped less experienced agents most, and made customers less likely to ask for a manager.
Its gains were largest on moderately rare problems, which is where a team’s own macros help least.
This is the wrong choice before the knowledge base is maintained, since the assistant will draft fluent versions of stale answers.
It is also the wrong choice for a team that will judge it on speed alone, since the same study found a small fall in quality for its most experienced agents.
Strengths for this problem
- Has the strongest field evidence of any AI option in support, with a 15% average rise in issues resolved per hour across 5,172 agents at a business software company.
- Helps new and less experienced agents most, shortening the months it takes them to handle technical tickets alone.
- Keeps an agent between the draft and the customer, so a wrong draft can be caught before it is sent.
- Draws on the same knowledge base the team already maintains, and shows which article each suggestion came from.
Limitations for this problem
- Helped the most experienced agents least, with small declines in their quality in the same study.
- Drafts are only as good as the articles and past tickets retrieved, and past tickets can contain answers that were wrong or are now out of date.
- Agents under time pressure may send drafts without reading them closely, which removes the safeguard the design depends on.
Data needed
- The live conversation, with the customer's plan, product version, configuration, and recent errors.
- Help centre and internal articles with product version and review date, and resolved tickets judged good by quality review.
- Which drafts agents sent unchanged, edited, or discarded, and the satisfaction and reopen outcome of each ticket.
How it is measured
- Resolution time and issues resolved per hour, compared with a group of agents without the assistant, by experience level.
- Customer satisfaction, reopen rate, and quality review scores for assisted conversations, by experience level.
- Share of drafts sent unchanged, edited, or discarded.
- Time for new agents to reach the team's standard resolution rate.
Pitfalls
- Retrieving from all past tickets, including wrong answers and answers for old versions, rather than from reviewed ones.
- Rolling out to every agent at once with no comparison group, so any effect on quality cannot be measured.
- Measuring speed only, when the evidence shows quality can fall for the most experienced agents.
- Pasting customers' credentials or private data into a third-party model without redaction or a data processing agreement.
Human oversight
The support agent reads, edits, and decides to send every reply. Team leads review a sample of assisted conversations each week and watch quality scores by experience level. The knowledge manager decides which articles and past tickets the assistant may retrieve from.