Asteria: Semantic-Aware Cross-Region Caching for Agentic LLM Tool Access

Abstract: LLM agents tackle data-intensive tasks such as deep research and code generation. However, their effectiveness depends on frequent interactions with knowledge sources across remote clouds or regions. Such interactions can create non-trivial latency and cost bottlenecks. Existing caching solutions focus on exact-match queries, limiting their effectiveness for semantic knowledge reuse. To address this challenge, we introduce Asteria, a novel cross-region knowledge caching architecture for LLM agents. At its core are two abstractions: Semantic Element (SE) and Semantic Retrieval Index (Sine). A semantic element captures the semantic embedding representation of an LLM query together with performance-aware metadata such as latency, cost, and staticity. Sine then provides two-stage retrieval: a vector similar index with semantic embedding for fast candidate selection and a lightweight LLM-powered semantic judger for precise validation. Atop these primitives, Asteria builds a new cache interface that includes a new semantic-aware cache hit definition, a cost-efficient eviction policy, and proactive prefetching. To reduce overhead, Asteria co-locates the small LLM judger with the main LLM using adaptive scheduling and resource sharing. Our evaluation demonstrates that Asteria delivers substantial performance improvements without compromising correctness. On representative search workloads, Asteria achieves up to a 3.6$\times$ increase in throughput by maintaining cache hit rates of over 85%, while preserving accuracy virtually identical to non-cached baselines. Asteria also improves throughput for complex coding tasks by 20%, showcasing its versatility across diverse agentic workloads.

• The paper presents Asteria, a semantic-aware caching system that reduces remote call latency from 1.08 s to 0.61 s and slashes external API calls by over 90%.
• The paper introduces a two-stage retrieval pipeline combining ANN search and an LLM-based semantic judger, achieving up to 3.6× higher throughput with hit rates exceeding 85%.
• The paper leverages resource-efficient co-location with an 80/20 GPU split and an LCFU eviction policy, delivering 6× more throughput per dollar while managing burst traffic.

Semantic-Aware Cross-Region Caching for Agentic LLM Tool Access: Asteria

Introduction and Motivation

Asteria addresses the critical latency and cost bottlenecks in agentic LLM deployments that rely on frequent remote tool calls, such as web search APIs and retrieval-augmented generation (RAG) backends. Unlike non-agentic LLM inference, agentic workflows are dominated by iterative "think-act-observe" loops, where each step may require external data retrieval. This introduces substantial cross-region latency (300–500 ms per call) and significant financial cost, especially at scale (e.g., $1.5–4.5M/month for 5–10M daily queries). Existing caching solutions—transformer KV caches, semantic prompt caches, and traditional storage caches—are inadequate for agentic workloads due to their reliance on exact-match or superficial embedding similarity, lack of semantic validation, and inability to handle rate limits or cross-region cost dynamics.

Figure 1: Agent-based application architecture.

Workload Analysis: Locality and Temporal Dynamics

Empirical analysis of search and code agent workloads reveals highly skewed, Zipfian access patterns, with a small set of "hot" queries or files dominating traffic. Google Trends data and code repository access logs both exhibit this property, indicating strong potential for caching. Additionally, search queries are bursty and temporally correlated, often spiking in response to external events. These characteristics necessitate a cache that is both semantically aware and adaptive to temporal dynamics.

Figure 2: Zipfian distribution in Google Trends data for top search topics.

Figure 3: Bursty and correlated search patterns driven by external events.

Asteria Architecture

Asteria introduces two core abstractions: the Semantic Element (SE) and the Semantic Retrieval Index (Sine). An SE encapsulates the agent's query, tool interaction, and retrieved response, augmented with metadata such as latency, cost, staticity, and access frequency. Sine implements a two-stage retrieval pipeline: (1) Approximate Nearest Neighbor (ANN) search for high-recall candidate selection, and (2) a lightweight LLM-based Semantic Judger for precise validation. This design transforms noisy similarity scores into reliable cache hit signals, ensuring only contextually appropriate and semantically equivalent results are served.

Figure 4: Asteria's agentic caching architecture with geo-distributed agent and data source.

Figure 5: Structure of Asteria's Semantic Element, including query, result, and metadata.

Semantic Retrieval and Cache Management

The retrieval process is formalized as a constrained optimization problem, balancing latency and semantic precision. ANN similarity threshold ($\tau_{\text{sim}}$) controls candidate recall, while the Semantic Judger threshold ($\tau_{\text{lsm}}$) enforces precision. Periodic offline recalibration of $\tau_{\text{lsm}}$ maintains target accuracy under workload drift. Cache management employs a Least Cost-Efficient and Frequently Used (LCFU) eviction policy, prioritizing items that maximize time/money savings per byte, and integrates a TTL-based aging mechanism to prevent staleness. Predictive prefetching leverages Markov models to anticipate future queries based on observed transitions, further reducing miss latency.

Figure 6: Priority-aware scheduling for agent and judger co-location on a single GPU.

Resource-Efficient Co-location

Asteria achieves resource efficiency by co-locating the agent LLM and the semantic judger on a single GPU, using CUDA MPS for asymmetric partitioning (e.g., 80/20 split) and a dynamic priority scheduler. This ensures that latency-critical agent tasks are never blocked by judger validation, maintaining high throughput and low cost without additional hardware.

Experimental Results

Throughput and Hit Rate

Asteria demonstrates substantial throughput improvements across diverse workloads. On skewed search benchmarks, it achieves up to 3.6× higher throughput than exact-match caching, with cache hit rates exceeding 85%. On bursty, trend-driven workloads, hit rates approach 95%, absorbing traffic spikes efficiently. In code generation tasks (SWE-Bench), Asteria delivers a 20% throughput boost, with a 45% hit rate, validating its generalizability.

Figure 7: Throughput on skewed search workload under varying cache ratios.

Figure 8: Throughput on trend-driven workload under varying cache ratios.

Figure 9: Throughput on SWE-Bench code workload under varying cache ratios.

Scalability and Latency

Asteria scales linearly with concurrency up to hardware limits, achieving up to 5.7× throughput improvement over baselines. Latency analysis shows that Asteria reduces per-request latency from 1.08 s to 0.61 s by replacing remote calls with fast local cache checks (0.05 s overhead).

Figure 10: Throughput under varying concurrency on Musique dataset.

Figure 11: Per-request end-to-end performance breakdown.

API Rate Limits and Cost Efficiency

By reducing external API calls by >90%, Asteria virtually eliminates rate-limit-induced throttling, increasing throughput by 4.2× under realistic rate limits. Cost analysis shows that Asteria delivers 6× more throughput per dollar than vanilla agentic systems, with co-location retaining 94% of dedicated throughput at half the hardware cost.

Figure 12: Data retrieval call and retries ratio, showing dramatic reduction in API calls and retries.

Accuracy

Naive semantic caching (ANN-only) suffers significant accuracy drops (e.g., 0.69 vs. 0.79 Exact Match on StrategyQA), but Asteria's judger maintains accuracy virtually identical to non-cached baselines, confirming the necessity of semantic validation.

Figure 13: Generation quality comparison between Asteria and vanilla Search-R1.

Policy and Implementation Studies

LCFU eviction policy outperforms LRU/LFU in system-wide latency reduction, prioritizing expensive-to-retrieve items. Co-location studies confirm near-baseline efficiency with minimal latency overhead. Recalibration overhead is negligible (<2% throughput loss).

Figure 14: The benefits brought by LCFU policy in cache hit rate and throughput.

Related Work

Asteria advances beyond semantic prompt caches [bang2023gptcache], which focus on inference optimization and lack semantic validation, and traditional distributed caches, which are limited to exact-match keys. It also differs from agent memory systems, which enhance reasoning but do not address execution-layer cost. Asteria uniquely targets the external tool call layer, integrating semantic-aware matching, adaptive policies, and resource-efficient deployment.

Implications and Future Directions

Asteria's architecture demonstrates that semantic-aware caching at the knowledge/tool boundary is essential for scalable, cost-effective agentic LLM deployments. The two-stage retrieval pipeline, adaptive cache management, and resource-efficient co-location collectively enable high throughput, low latency, and financial sustainability without sacrificing correctness. Future work may explore tighter integration with agent memory systems, more sophisticated temporal models for prefetching, and further hardware optimizations for multi-agent scenarios.

Conclusion

Asteria provides a robust solution to the latency and cost challenges of agentic LLM tool access, combining semantic matching, adaptive cache management, and efficient resource sharing. Its empirical results validate strong performance and accuracy, establishing a new standard for semantic-aware cross-region caching in agentic AI systems.