COTA: Improving the Speed and Accuracy of Customer Support through Ranking and Deep Networks

Abstract: For a company looking to provide delightful user experiences, it is of paramount importance to take care of any customer issues. This paper proposes COTA, a system to improve speed and reliability of customer support for end users through automated ticket classification and answers selection for support representatives. Two machine learning and natural language processing techniques are demonstrated: one relying on feature engineering (COTA v1) and the other exploiting raw signals through deep learning architectures (COTA v2). COTA v1 employs a new approach that converts the multi-classification task into a ranking problem, demonstrating significantly better performance in the case of thousands of classes. For COTA v2, we propose an Encoder-Combiner-Decoder, a novel deep learning architecture that allows for heterogeneous input and output feature types and injection of prior knowledge through network architecture choices. This paper compares these models and their variants on the task of ticket classification and answer selection, showing model COTA v2 outperforms COTA v1, and analyzes their inner workings and shortcomings. Finally, an A/B test is conducted in a production setting validating the real-world impact of COTA in reducing issue resolution time by 10 percent without reducing customer satisfaction.

• The paper presents COTA, which improves ticket classification and response selection by reformulating multi-class challenges as ranking problems.
• It introduces an Encoder-Combiner-Decoder (ECD) architecture that integrates heterogeneous features and enables effective multi-task learning.
• Empirical results show that COTA v2 outperforms v1, delivering a significant accuracy boost and reducing average ticket handling time by around 10%.

COTA: Deep Learning-Driven Customer Support Automation

Introduction

The paper "COTA: Improving the Speed and Accuracy of Customer Support through Ranking and Deep Networks" (1807.01337) presents an empirical investigation and systemization of machine learning and deep learning solutions for automating customer support ticket classification and response selection in high-volume settings. The COTA system, deployed at Uber, leverages both traditional feature-engineered ML pipelines (COTA v1) and an extensible deep learning framework (COTA v2). The study systematically compares these approaches, introduces a novel Encoder-Combiner-Decoder (ECD) architecture for multi-task learning with heterogeneous features, and analyzes both offline and online business metrics, including latency and customer satisfaction.

System Design and Problem Formulation

COTA targets two primary tasks in CSR workflows: (1) contact type identification (analogous to intent detection) and (2) automated reply template selection. Both tasks present high-cardinality multi-class problems, impacted by vast numbers of issue types and response templates, compounded by class imbalance and hierarchical dependencies among contact types.

COTA v1: Feature Engineering and Ranking

In COTA v1, the authors operationalize these challenges by transforming classical multi-class classification into a pairwise ranking problem using extensive feature engineering:

• Text preprocesssing: Tokenization, lemmatization, stop-word removal, bag-of-words, TF-IDF, and LSA projections (Truncated-SVD) to extract semantic vectors.
• Similarity-based Features: Prototypes for each class (contact type, reply template) are distilled; incoming tickets are scored via cosine similarity in semantic space, producing high-signal features for ranking.
• Algorithms: Both multi-class RandomForest and pointwise-ranking with binary relevance labeling are employed. Ranking, leveraging similarity features, yields clear empirical advantages, particularly in data with thousands of long-tail classes.

COTA v2: Encoder-Combiner-Decoder Deep Learning

COTA v2 is architected for the multi-modal, multi-task nature of support tickets:

• Encoder: Each input feature (raw text, categorical, numerical, binary) is processed by a task-specific encoder—character/word CNN, RNN, or hybrids for text; embeddings for categorical features.
• Combiner: Feature vectors are concatenated and optionally further processed by dense layers, acting as a nonlinear aggregator.
• Decoder: Predictors (heads) are instantiated per task, e.g., softmax for categorical outputs, MSE for regression. Decoders can be interdependent, e.g., reply template decoder conditions on contact type prediction.

Importantly, the ECD architecture encodes prior knowledge by: - Predicting contact type hierarchy paths (sequence decoder outputting nodes from root to leaf). - Structuring decoder dependencies reflecting workflow logic (reply template depends on contact type outcome).

Multi-task training is implemented via a weighted sum of task-specific losses, allowing joint optimization.

Empirical Evaluation

Dataset

Over 3 million historical customer support tickets from Uber serve as the experimental corpus. Feature heterogeneity is substantial, with long-tailed distributions for both contact types (7-level hierarchy) and templates.

Model Selection and Hyperparameter Optimization

Extensive grid/random search are conducted for both ML (RandomForest) and DL (encoders, combiner, decoders, optimizer configuration). Notably, Word-CNN provides the best speed-accuracy tradeoff for textual encoding in COTA v2.

Results

COTA v1: Ranking Surpasses Classification

In both contact type and reply template prediction, the ranking formulation dramatically outperforms direct classification. Accuracy gains are pronounced, especially for reply template selection (+11% accuracy, +19% Hits@3). Combined accuracy across both tasks improves by roughly 14%.

COTA v2: Sequential Decoders and Dependency Injection

Utilizing a sequential decoder for contact type tree traversal yields higher accuracy and more "reasonable" errors (parent nodes favored over unrelated types), reflected in a 6% improvement when parent nodes are credited. Conditioning the reply template head on predicted contact type produces clear enhancements in both single-task and joint-task metrics (reply accuracy, combined accuracy +13%). These findings directly support the hypothesis that architectural priors and dependency tracking enhance system performance and output coherence.

COTA v1 vs. COTA v2

COTA v2 robustly surpasses COTA v1 by large margins: +16% contact type accuracy, +8% overall multi-task accuracy. This affirms the advantages of deep, multi-modal, multi-task networks under large-scale, high-noise, and high-cardinality regimes.

Model Introspection and Error Analysis

t-SNE projections of learned word and class embeddings demonstrate that the model effectively learns semantic similarity (e.g., "car" and "vehicle" cluster; driver/rider contact types are separated), supporting the claim that ECD-based networks internalize domain structure. Class imbalance remains a limiting factor, with F1 scores closely tracking class frequency; the authors suggest class pruning and metadata enrichment for future work.

Business Metrics and A/B Testing

A fielded RCT involving thousands of CSRs and real-world tickets demonstrates that introduction of COTA reduces average ticket handling time by ~10% (statistically significant), with preservation—and occasional enhancement—of customer satisfaction. These production results validate the offline metrics and indicate meaningful impact on core operational KPIs.

Implications and Future Directions

The contributions of this work are both architectural and procedural. COTA's ECD formalism enables plug-and-play handling of heterogeneous features, scalability to thousands of classes, and seamless multi-task optimization with explicit task dependencies. The demonstrated gains from architecture-informed priors (hierarchy traversal, decoder dependency) advocate for domain knowledge inclusion in end-to-end neural models, especially in enterprise scenarios with complex workflows.

Practical deployments such as COTA illustrate the readiness of deep learning techniques for mission-critical, high-throughput support environments, with the caveat that class imbalance and rare event generalization need further innovation. The open sourcing of the implementation (ludwig) promises increased adoption and benchmarking for similar use cases.

Expected advancements include research on few-shot learning for rare issue types, dynamic template generation, continual learning from CSR feedback, and exploration of more sophisticated hierarchical/graph neural network decoders.

Conclusion

This paper establishes a rigorous baseline for automated customer support decision support systems, demonstrating that the combination of ranking-based ML and attention to domain-specific structure via deep, multi-task networks can yield substantial performance and efficiency improvements. The COTA system’s empirical validation—through both offline metrics and online business impact—underscores the feasibility and necessity of integrating advanced machine learning into customer operations platforms. Future progress hinges on the integration of architectural priors, handling of rare events, and richer human-in-the-loop learning paradigms (1807.01337).