HMI: Hierarchical Knowledge Management for Efficient Multi-Tenant Inference in Pretrained Language Models

Abstract: The significant computational demands of pretrained LLMs (PLMs), which often require dedicated hardware, present a substantial challenge in serving them efficiently, especially in multi-tenant environments. To address this, we introduce HMI, a Hierarchical knowledge management-based Multi-tenant Inference system, designed to manage tenants with distinct PLMs resource-efficiently. Our approach is three-fold: Firstly, we categorize PLM knowledge into general, domain-specific, and task-specific. Leveraging insights on knowledge acquisition across different model layers, we construct hierarchical PLMs (hPLMs) by extracting and storing knowledge at different levels, significantly reducing GPU memory usage per tenant. Secondly, we establish hierarchical knowledge management for hPLMs generated by various tenants in HMI. We manage domain-specific knowledge with acceptable storage increases by constructing and updating domain-specific knowledge trees based on frequency. We manage task-specific knowledge within limited GPU memory through parameter swapping. Finally, we propose system optimizations to enhance resource utilization and inference throughput. These include fine-grained pipelining via hierarchical knowledge prefetching to overlap CPU and I/O operations with GPU computations, and optimizing parallel implementations with batched matrix multiplications. Our experimental results demonstrate that the proposed HMI can efficiently serve up to 10,000 hPLMs (hBERTs and hGPTs) on a single GPU, with only a negligible compromise in accuracy.

Hierarchical Knowledge Management for Efficient Multi-Tenant Inference in Pretrained Language Models

The paper titled "HMI: Hierarchical Knowledge Management for Efficient Multi-Tenant Inference in Pretrained Language Models" addresses the challenges associated with serving a large number of pretrained language models (PLMs) efficiently, particularly in multi-tenant environments where multiple models are required to operate simultaneously on shared hardware resources. The authors propose the Hierarchical Knowledge Management-based Multi-tenant Inference system (HMI), which demonstrates a novel approach to manage the computational demands of PLMs, aiming to optimize inference processes and resource utilization.

Key Contributions

The paper introduces HMI, which effectively organizes the hierarchical knowledge derived from PLMs into general, domain-specific, and task-specific categories. It tackles the problem of computationally intensive inference by categorizing and managing PLM knowledge differently across various levels:

1. Hierarchical PLM Construction: HMI constructs hierarchical PLMs (\textsf{hPLM}s) by separating general, domain-specific, and task-specific knowledge, thus optimizing resource utilization. General knowledge is derived from the pretrained model, while domain-specific and task-specific knowledge is acquired during further pretraining and fine-tuning processes, respectively. By extracting and decoupling these different types of knowledge, HMI significantly reduces GPU memory consumption for each tenant.

2. Management of Hierarchical Knowledge: The system employs a frequency-based strategy to manage domain-specific knowledge, which is stored in precomputed lookup tables (PLOT). Task-specific knowledge is handled via adapters, with parameters swapped as needed during inference. This method allows for maintaining domain-specific knowledge efficiently while minimizing additional storage overhead and managing task-specific models within limited GPU memory.

3. System Optimizations: The authors employ several optimizations to enhance throughput and resource utilization. These include pipelined hierarchical knowledge prefetching to overlap CPU operations with GPU computations and batched matrix multiplications to enable efficient parallel inference implementation. As a result, HMI can serve massive numbers of \textsf{hPLM}s concurrently without sacrificing accuracy.

Implications and Future Directions

The paper presents several strong numerical results demonstrating that HMI can support up to 10,000 \textsf{hPLM}s on a single GPU with minimal accuracy loss. This significant achievement implies that massive multi-tenancy inference can be achieved on existing cloud infrastructure without the need for substantial computational resource expansion.

The implications of this work suggest practical advancements in how cloud platforms can manage computational and memory resources for large-scale PLM deployments across varied applications, potentially improving efficiency in hosting platforms and reducing operational costs.

The framework and methodologies introduced provide a basis for exploring further developments in optimizing other types and architectures of language models, including generative models such as GPT variants. Incorporating speculative decoding or alternative mechanisms for optimizing transformer architectures could further enhance efficiency and scalability.

In conclusion, the hierarchical organization and management of PLM knowledge as posited by HMI offer an effective avenue for improving multi-tenant inference performance under constrained resources. Future research could leverage these insights to explore even more efficient methods for model deployment across diverse AI applications.