Hierarchical Long-Term Semantic Memory for LinkedIn's Hiring Agent

Abstract: LLM agents are increasingly used in real-world products, where personalized and context-aware user interactions are essential. A central enabler of such capabilities is the agent's long-term semantic memory system, which extracts implicit and explicit signals from noisy longitudinal behavioral data, stores them in a structured form, and supports low-latency retrieval. Building industrial-grade long-term memory for LLM agents raises five challenges: scalability, low-latency retrieval, privacy constraints, cross-domain generalizability, and observability. We introduce the Hierarchical Long-Term Semantic Memory (HLTM) framework, which organizes textual data into a schema-aligned memory tree that captures semantic knowledge at multiple levels of granularity, enabling scalable ingestion, privacy-aware storage, low-latency retrieval, and transparent provenance; HLTM further incorporates an adaptation mechanism to generalize across diverse use cases. Extensive evaluations on LinkedIn's Hiring Assistant show that HLTM improves answer correctness and retrieval F1 significantly by more than 10%, while significantly advancing the Pareto frontier between query and indexing latency. HLTM has been deployed in LinkedIn's Hiring Assistant to power core personalization features in production hiring workflows.

• The paper introduces HLTM, a hierarchical memory framework that aligns with business schemas to enhance retrieval accuracy and enforce privacy.
• It employs multi-view representations—facets, QA pairs, and summaries—to enable lossless aggregation and efficient incremental updates.
• Experimental results show HLTM outperforms baselines by over 15% in correctness and reduces online latency, validating its enterprise-scale effectiveness.

Hierarchical Long-Term Semantic Memory for LLM Agents at Enterprise Scale

Introduction and Motivation

The deployment of LLM-powered agents in production systems, particularly those requiring user- and context-specific personalization, exposes the inadequacy of standard context window-driven memory protocols. Effective industrial usage requires agentic memory systems capable of both extracting implicit/explicit signals from noisy, behavioral data and supporting privacy-bound, low-latency retrieval. "Hierarchical Long-Term Semantic Memory for LinkedIn's Hiring Agent" (2604.26197) proposes HLTM, a schema-aligned hierarchical tree memory framework, explicitly targeting the constraints and operational requirements of large-scale enterprise LLM agents.

HLTM Design: Schema-Aligned Hierarchical Memory

HLTM deviates from conventional semantic clustering or learned-memory layouts by grounding its hierarchy in the business schema (e.g., projects, recruiter seats, organization), thereby aligning the physical memory topology with privacy and access control domains.

Figure 1: Overview of Hierarchical Long-Term Semantic Memory (HLTM).

Each node aggregates downstream leaves—fine-grained units (e.g., hiring projects) compose seat-level recruiter nodes, which in turn roll up to organization-level roots. This structural grounding enables lossless, localized incremental updates, robust provenance, and strict isolation under privacy constraints.

Distinct multi-view representations are maintained at each node:

• Facets: key–value pairs extracted as salient, interpretable facts, embedded for rapid schema-aware matching.
• QA pairs: offline-generated, answerable questions closely tied to the typical form of user queries, improving semantic retrieval.
• Summaries: natural-language compressions for cases requiring unstructured aggregation or synthesis, with both detailed and single-sentence variants used for retrieval.

Offline and Online Operations: Aggregation, Indexing, Retrieval

Sequential, bottom-up aggregation composes memory from leaves to root—parent nodes recursively synthesize and resolve child memories using LLM prompting, generating higher-order facts while reconciling low-salience and contradictory signals.

Lossless, nearline incremental indexing is central to scalability. Only updated leaves and their ancestor chains require recomputation, with unaffected subtrees left invariant. This preserves the semantics of a fresh global rebuild with substantially reduced computational cost and update frequency.

Figure 2: Incremental HLTM updates only require re-indexing paths from modified leaves, minimizing unnecessary computation.

At retrieval, HLTM enforces identity-scoped subtree filtering: retrieval is strictly bounded to the subtree implied by the query's access domain. A hybrid multi-signal retriever evaluates facet, QA, and summary relevance independently, aggregating diverse matches to maximize coverage across query types (structured, natural, and open-ended).

Integration with LinkedIn's Hiring Assistant

HLTM powers LinkedIn's Hiring Assistant, an LLM-driven agent supporting recruiter workflows with persistent, privacy-aligned, and adaptive long-term memory. The hiring process (project creation, candidate search, outreach) generates records which HLTM absorbs as fine-grained leaves and propagates up to recruiter- and account-level memories. When recruiters articulate new requirements or queries, the Hiring Assistant leverages HLTM for real-time retrieval and provenance-tracked response synthesis.

Figure 3: HLTM supports natural language queries to distill recruiter preferences and update hiring requirements.

Experimental Evaluation

Extensive experiments benchmark HLTM against retrieval baselines (RAG, full-context) and recent advanced memory systems (A-Mem, GraphRAG, Mem0, SimpleMem, ReadAgent, RAPTOR, HippoRAG) across summary and retrieval-style queries over a curated, human-annotated dataset derived from LinkedIn Hiring Assistant logs.

On correctness and token-level F1 for summary-style queries, HLTM achieves 0.798 correctness and 0.635 F1, outperforming the best baseline (Conventional RAG, HippoRAG) by more than 15%. Retrieval queries yield a 0.712 F1, an improvement of more than 10% over baselines. Particularly notable is that HLTM achieves this with lower online query latency and substantially reduced token usage for the answer-generation context.

Figure 4: HLTM provides higher answer correctness at lower latency versus all baseline methods.

Latency analysis demonstrates that HLTM advances the quality–latency Pareto frontier: highly parallel index construction and retrieval-collapse design minimize both offline and online latency, a core requirement for production agent systems with high data velocity.

Figure 5: Query-indexing latency comparison—HLTM achieves strong trade-offs and efficiency across operational metrics.

Ablation Studies and Robustness

Ablation analysis isolates the necessity of hierarchical aggregation, adaptation, and each view type. Notably, disabling tree aggregation results in a severe correctness decline for summaries (–12 points) and retrieval-style F1 (–18 points), indicating the criticality of cross-leaf synthesis. Adaptation via query-pattern analysis further boosts performance, confirming the importance of matching memory optimization to workload characteristics rather than relying on undirected LLM aggregation.

Hyperparameter sweeps show that HLTM is robust to the retrieval pool size $k$: performance quickly plateaus at small $k$, avoiding the confusion/recall trade-off common in flat or leaf-only systems.

Figure 6: Retrieval F1 plateaus at small $k$ in HLTM, indicating robustness and low hyperparameter sensitivity.

Privacy, Provenance, and Observability

By hardwiring the memory partitioning to business entities, HLTM enforces strict privacy policies, prevents data drift across access domains, and supports auditable provenance—all mission-critical in enterprise, multi-tenant environments. Retrieved contexts carry stable business identifiers, enabling transparent debugging, regulatory compliance (e.g., GDPR), and source-of-truth synchronization.

Practical and Theoretical Implications

HLTM demonstrates that highly-structured, schema-aligned memory systems deliver substantial gains in answer quality, latency, and privacy over purely semantic or loosely organized long-context methods. This architectural approach underscores the benefits of fusing enterprise data models with agentic memory, particularly in regulated, large-scale, and mission-critical deployments.

Future developments should explore richer schema compositions (cross-entity and multimodal aggregation), deeper integration with agent planners/controllers, and generalization to domains beyond hiring—where transparent, fast, and privacy-respecting LLM agent memory is mandatory.

Conclusion

HLTM operationalizes hierarchical, multi-view long-term memory for industrial LLM agents, achieving substantial improvements in answer correctness, retrieval F1, and latency compared to contemporary baselines, while satisfying privacy and observability obligations (2604.26197). The schema-aligned design, adaptive aggregation, and incremental indexing provide a strong foundation for deploying robust, enterprise-scale agentic memory in practical applications, as exemplified by its integration with LinkedIn’s Hiring Assistant.