[1911.02557v1] Feedback-Based Self-Learning in Large-Scale Conversational AI Agents
In this paper, we presented a self-learning system that is able to efficiently target and rectify both systemic and customer errors at runtime by means of query reformulation
Abstract Today, most of the large-scale conversational AI agents such as Alexa, Siri, or Google Assistant are built using manually annotated data to train the different components of the system including Automatic Speech Recognition (ASR), Natural Language Understanding (NLU) and Entity Resolution (ER). Typically, the accuracy of the machine learning models in these components are improved by manually transcribing and annotating data. As the scope of these systems increase to cover more scenarios and domains, manual annotation to improve the accuracy of these components becomes prohibitively costly and time consuming. In this paper, we propose a system that leverages customer/system interaction feedback signals to automate learning without any manual annotation. Users of these systems tend to modify a previous query in hopes of fixing an error in the previous turn to get the right results. These reformulations, which are often preceded by defective experiences caused by either errors in ASR, NLU, ER or the application. In some cases, users may not properly formulate their requests (e.g. providing partial title of a song), but gleaning across a wider pool of users and sessions reveals the underlying recurrent patterns. Our proposed self-learning system automatically detects the errors, generate reformulations and deploys fixes to the runtime system to correct different types of errors occurring in different components of the system. In particular, we propose leveraging an absorbing Markov Chain model as a collaborative filtering mechanism in a novel attempt to mine these patterns. We show that our approach is highly scalable, and able to learn reformulations that reduce Alexa-user errors by pooling anonymized data across millions of customers. The proposed self-learning system achieves a win/loss ratio of 11.8 and effectively reduces the defect rate by more than 30% on utterance level reformulations in our production A/B tests. To the best of our knowledge, this is the first self-learning large-scale conversational AI system in production.
‹Figure 1: A high-level overview of the deployed architecture with our reformulation engine in context of the overall system in (a) and the offline sub-system that updates its online counterpart on a daily cadence. (Introduction)Figure 2: A visual representation of the Markov model constructed in the interpretation space, H, over three separate sessions, (a), (b), and (c), of users attempting to play the album ”Despicable Me”, and how solving for the path with the highest likelihood of success, (+), given by the darkened edged in (d), can allow for the defective utterances to be reformulated into a more successful query, as summarized in (e). Note that here, for demonstration purposes, we only show 3 interactions. However, in practice, we had a higher threshold for the minimum number of customers and interactions to have better estimates for the probabilities. (Absorbing Markov Chain)
