Memory Systems, AI Agents and LLMs

A Unified Framework for Strategy & Silicon

My previous article will serve as a reference point from which this one builds. Due to the high volume of research releases, this will be written and continue, in near note form. The repository of reference (helpful to me) is segregated to be hopefully useful for relevant applications, as you deem fit. Consider this a Part II to Machine Memory: The Math That Matters:

Combined, both explore the evolution of AI memory architectures, moving beyond the initial "vector gold rush" toward more hardware-aware and adaptive systems. They establish fundamental economic, geometric, and architectural laws that highlight the high costs and mathematical limitations of traditional vector searches. To address these constraints, I had highlighted and introduced MemEvolve, a meta-evolutionary engine that automates the design of memory modules by learning from performance failures.

Though an exciting and ever-evolving space, a comprehensive layered taxonomy is presented, mapping out a research landscape that spans from physical hardware infrastructure to complex cognitive architectures. This integrated approach encourages developers and planners to balance computational efficiency with task-specific needs, such as managing the "memory wall" through heterogeneous storage hierarchies.

Ultimately, and I hope, logically, to argue that the future of AI depends on adaptive learning strategies and structured memory rather than just increasing raw context size. This note also therefore refers:

Distinguishing Both Articles

At a high level, the two articles provide distinct but complementary perspectives on machine memory. While one focuses on the strategic and mathematical laws governing memory architecture, this article provides a comprehensive technical taxonomy centered on the hardware-software hierarchy.

Coverage and Contrast

Article 1 (”Machine Memory: The Math That Matters”)

• Strategic Focus: This article is oriented toward decision-making frameworks. It outlines five specific frameworks (Vector Economics, Welch Bound, Memory Triangle, Continuous Retrieval, and Context Maximalism) to help developers and planners guide choices of the right architecture for their specific constraints and timelines.

• Mathematical Limits: It emphasizes the “geometric ceiling” of vector spaces, specifically citing the Welch Bound, which explains why cramming too much data into a fixed vector space leads to “semantic collisions” and reduced precision.

• Economic Realism: It highlights a 20-80x cost difference between vector search and traditional keyword search (BM25), arguing for a “BM25 Renaissance” in large-scale systems.

• Meta-Learning: It introduces MemEvolve, a system that uses “dual evolution” to automatically navigate the trade-offs between performance, cost, and delay by evolving the memory architecture itself based on task feedback.

Article 2, This Article (”Memory Systems AI Agents and LLMs”)

• Infrastructural Focus: This article focuses more on a layered taxonomy (Layer 0 to Layer 7), with a heavy emphasis on Layer 0: Hardware & System Infrastructure.

• Hardware Bottlenecks: It dives deeply into the “Memory Wall,” discussing GPU VRAM capacity, bandwidth limitations, and the necessity of “Processing-in-Memory” (PIM) and “In-Storage Computing”.

• Comprehensive Taxonomy: It categorizes 84 research papers across eight functional layers, including KV cache efficiency, multimodal memory, and tool integration.

• Implementation Guides: It provides (I hope) a practical guide for deploying agents on different hardware tiers, such as mobile/edge (6GB RAM) versus enterprise (8x80GB GPU) setups.

---

To wit, to understand the difference between these two articles, imagine you are building a professional kitchen. Article 1 is the Executive Chef’s Strategy, focusing on how to balance the menu (the Memory Triangle) and when to buy pre-chopped ingredients versus fresh ones to save money (Vector Economics). Article 2 is the Kitchen Engineer’s Blueprint, focusing on how many ovens the electrical grid can support, where the pipes are located, and how to keep the refrigerators from overheating (Hardware Infrastructure).

---

Unified Framework: Hardware Foundation + Memory Solutions

This framework integrates memory architectures (what types exist) with bottleneck solutions (what problems they solve), creating a comprehensive map of the research landscape.

---

Framework Structure

BOTTLENECK LAYER → MEMORY TYPE SOLUTION → SPECIFIC IMPLEMENTATIONS

Layer 0: Hardware & System Infrastructure ⭐ NEW

Bottleneck: The “Memory Wall”

Problem: GPU memory capacity, bandwidth, and cost constrain model size, context length, and deployment scalability.

---

0.1 Hardware-Software Co-Design & Processing-in-Memory (PIM)

Hardware-Software Co-Design & PIM

1. L3: DIMM-PIM Integrated Architecture and Coordination for Scalable Long-Context LLM Inference (2024)

2. PIMphony: Overcoming Bandwidth and Capacity Inefficiency in PIM-based Long-Context LLM Inference System (2024)

3. HPU: High-Bandwidth Processing Unit for Scalable, Cost-effective LLM Inference via GPU Co-processing (2024)

4. Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Design (2024)

5. InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference (2024)

Key Innovation: Moving computation to where data resides, rather than moving data to computation.

---

0.2 Model Offloading & Heterogeneous Memory Hierarchies

Model Offloading & Heterogeneous Memory

6. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU (2023)

7. Scaling Up On-Device LLMs via Active-Weight Swapping Between DRAM and Flash (2024)

8. Hardware-based Heterogeneous Memory Management for Large Language Model Inference (2024)

9. ZeRO-Offload: Democratizing Billion-Scale Model Training (2021)

10. DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale (2022)

Key Innovation: Treating memory as a hierarchy (GPU VRAM → CPU DRAM → SSD/Flash) with intelligent orchestration.

---

0.3 Inference System Optimizations

Inference System Optimizations

11. Jenga: Effective Memory Management for Serving LLM with Heterogeneity (2024)

12. Combating the Memory Walls: Optimization Pathways for Long-Context Agentic LLM Inference (2024)

13. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (2024)

14. Orca: A Distributed Serving System for Transformer-Based Generative Models (2022)

Key Innovation: System-level orchestration that maximizes existing hardware capabilities.

---

0.4 Hardware Analysis & Future Directions

Hardware Analysis

15. Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures (2024)

16. Counting Carbon: A Survey of Factors Influencing the Emissions of Machine Learning (2023)

---

Layer 1: Hardware & Computational Efficiency

Bottleneck: KV Cache Memory Consumption

Problem: GPU memory grows linearly with sequence length, limiting batch size and context windows.

Layer 1: Computational Efficiency (KV Cache)

Quantization

17. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (2024)

18. Mixture-of-Memory: Reformulating Self-Attention with Mixture of Experts (2024)

Pruning & Eviction

19. SnapKV: LLM Knows What You are Looking for Before Generation (2024)

20. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models (2023)

Memory Management

21. Efficient Memory Management for Large Language Model Serving with PagedAttention (2023) - vLLM paper

22. SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Caching (2024)

Compression

23. AutoCompressor: Compressing Context into Summary Vectors (2023)

24. Sparseformer: Efficient Transformer via Learnable Sparsification (2021)

Survey Coverage:

• A Survey on Large Language Model Acceleration based on KV Cache Management (2024)

---

Layer 2: Context Window & Working Memory Constraints

Bottleneck: Limited Context Window

Problem: Fixed-size context causes “lost in the middle” and forces constant information eviction.

Layer 2: Context Window & Working Memory

Hierarchical Memory Management

25. MemGPT: Towards LLMs as Operating Systems (2023)

26. Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agents (2025)

27. SimpleMem: Efficient Lifelong Memory for LLM Agents (2025)

28. HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks (2024)

29. ReSum: Recursive Summarization for Long Context Language Models (2025)

Agentic Memory Systems

30. A-Mem: Adaptive Memory for Long-Context Language Understanding (2024)

31. G-Memory: Goal-Conditioned Memory for Autonomous Agents (2024)

Long-Context Processing

32. Mamba: Linear-Time Sequence Modeling with Selective State Spaces (2023)

33. Memformer: A Memory-Augmented Transformer for Sequence Modeling (2020)

34. MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression (2024)

35. NSA: Neighborhood Sliding Attention for Long Context (2024)

Few-Shot Prompting

36. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (2022)

37. PAL: Program-aided Language Models (2022)

Survey Coverage:

• A Survey of Context Engineering for Large Language Models (2024)

• Memory in the Age of AI Agents (2024)

---

Layer 3: Long-Term Persistence & Episodic Memory

Bottleneck: Information Persistence Across Sessions

Problem: No standardized way to store, retrieve, and evolve memories across episodes.

Layer 3: Long-Term Persistence & Episodic Memory

Self-Evolving Memory

38. Memento: Facilitating Long-Horizon Memory in LLM-based Agents (2023)

39. H2R: Human-in-the-Loop Hierarchical Memory for Long-Horizon Robotic Tasks (2024)

Modular RAG

40. FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research (2024)

41. ComposeRAG: Composable Retrieval-Augmented Generation (2024)

Graph RAG

42. LightRAG: Simple and Fast Retrieval-Augmented Generation (2024)

43. HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models (2024)

Agentic RAG

44. PlanRAG: A Plan-then-Retrieval Augmented Generation for Generative Large Language Models as Decision Makers (2024)

45. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (2023)

Infrastructure

46. MemOS: A Memory OS for AI System (2024)

47. MemoryBank: Enhancing Large Language Models with Long-Term Memory (2023)

Survey Coverage:

• Rethinking Memory in LLM based Agents (2024)

---

Layer 4: Procedural Memory & Skill Learning

Bottleneck: Cannot Learn “How To” from Experience

Problem: Agents rely on static prompts and don’t evolve procedural knowledge.

Layer 4: Procedural Memory & Skill Learning

Parametric Memory

48. Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization (2023)

49. Leveraging Early Experiences for Efficient Policy Learning in Multi-Agent Systems (2024)

RL-Enabled Memory

50. MemAgent: Towards Adaptive Memory Management for LLM-based Agents (2024)

51. RMM: Reinforced Memory Management for Class-Incremental Learning (2024)

52. MemSearcher: Reinforcement Learning Based Memory Searcher for Large Language Models (2023)

53. MEM1: Memory-Augmented Reinforcement Learning with Task-Specific Embedding (2024)

54. Memory-R1: Joint Training of Memory and Reasoning for Long-Horizon Tasks (2025)

Procedural Learning

55. Memp: Exploring Agent Procedural Memory (2024)

56. Real-Time Procedural Learning From Experience for AI Agents (2024)

Tool Learning

57. Toolformer: Language Models Can Teach Themselves to Use Tools (2023)

58. VTool-R1: Vision-Language Tool Reasoning and Execution (2025)

---

Layer 5: Multimodal & Sensory Memory

Bottleneck: Text-Only Memory is Insufficient

Problem: Real-world agents need visual, auditory, and sensory memories.

Layer 5: Multimodal & Sensory Memory

Vision-Language Memory

59. Ella: Efficient Large-Scale Vision-Language Representation Learning (2024)

60. ViloMem: Video-Language Memory for Long-Form Video Understanding (2024)

61. M3-Agent: Multi-task Multi-modal Agentic Framework (2024)

Latent Memory

62. MemoryLLM: Towards Self-Updatable Large Language Models (2024)

63. M+: Memory-Augmented Pre-training for Language Models (2024)

64. MemGen: Generative Memory for Long-Context Language Models (2024)

---

Layer 6: Tool Integration & Meta-Cognition

Bottleneck: Disconnected Tools and Reasoning

Problem: Tools, reasoning, and memory operate in silos.

Layer 6: Tool Integration & Meta-Cognition

Tool-Integrated Reasoning

65. ReTool: Retrieval-Augmented Tool Learning (2023)

66. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (2023)

Tool Selection

67. AutoTool: Automatic Tool Selection and Orchestration for LLM Agents (2024)

68. VisTA: Visual Tool Affordances for LLM Agents (2024)

Self-Reflection

69. Self-Refine: Iterative Refinement with Self-Feedback (2023)

70. CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023)

Communication Protocols

71. ANP: Agent Negotiation Protocol for Multi-Agent Systems (2024)

72. A2A: Agent-to-Agent Communication Framework (2024)

73. MCP: Model Context Protocol (2024)

74. Agora: Dynamic Multi-Agent Dialogue Systems (2023)

---

Layer 7: Cognitive Architecture & Meta-Memory

Bottleneck: Lack of Systematic Memory Design

Problem: No unified theory connecting memory representations, operations, and human cognition.

Layer 7: Cognitive Architecture & Meta-Memory

Memory Graph Structures

75. Zep: Long-Term Memory for AI Assistants (2023)

76. AriGraph: Learning Knowledge Graph World Models with Episodic Memory (2024)

Cognitive Foundations

77. AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agents (2024)

78. Memory Consolidation for Continual Learning in Neural Networks (2024)

Dynamic Memory Systems

79. General Agentic Memory Via Deep Research (2024)

Survey Coverage:

• A Survey on the Memory Mechanism of Large Language Model based Agents (2024)

• Memory in the Age of AI Agents (2024)

---

Cross-Cutting Infrastructure

Hardware-Level Optimization

Comprehensive Surveys

80. Memory in the Age of AI Agents (2024)

81. A Survey on Large Language Model Acceleration based on KV Cache Management (2024)

82. Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topics (2024)

83. A Survey of Context Engineering for Large Language Models (2024)

84. A Survey on the Memory Mechanism of Large Language Model based Agents (2024)

---

Quick Access by Problem Type

“I need to reduce GPU memory usage”

→ Layer 0: FlexGen, ActiveFlow, Jenga
→ Layer 1: KVQuant, PagedAttention, SnapKV

“I need longer context windows”

→ Layer 0: L3, PIMphony
→ Layer 1: SentenceKV, AutoCompressor
→ Layer 2: MemGPT, Agentic Memory

“I need persistent agent memory”

→ Layer 3: MemOS, LightRAG, HippoRAG
→ Layer 4: Memp, Real-Time Procedural Learning

“I need agents that learn skills”

→ Layer 4: MemAgent, Memory-R1, Toolformer
→ Layer 6: ToolLLM, AutoTool

“I need multimodal agents”

→ Layer 5: ViloMem, M3-Agent, MemoryLLM

“I want to understand the field”

→ Surveys: Memory in the Age of AI Agents, KV Cache Survey

Meta-Analysis: Memory Type × Bottleneck Matrix

Cross-Layer Integration Map

Layer 0: Hardware Infrastructure (NEW)
         ↓ Enables ↓
Layer 1: Computational Efficiency (KV Cache)
         ↓ Feeds into ↓
Layer 2: Context Window Management
         ↓ Supports ↓
Layer 3: Long-Term Persistence
         ↓ Enables ↓
Layer 4: Procedural Learning
         ↓ Integrates ↓
Layer 5: Multimodal Memory
         ↓ Coordinates ↓
Layer 6: Tool Integration
         ↓ Informs ↓
Layer 7: Cognitive Architecture

---

Hardware × Software Co-Design Examples

Example 1: Long-Context Inference Stack

Result: 10M+ token contexts on consumer hardware

---

Example 2: On-Device Mobile Agents

Result: 7B+ parameter agents on smartphones

---

Example 3: Data Center Agentic Systems

Result: Cost-effective multi-agent deployments

---

Hardware Bottleneck × Memory Type Solutions

Meta-Analysis: Memory Type × Bottleneck Matrix

Legend: ✓✓✓ Primary solution space | ✓✓ Secondary | ✓ Emerging | - Minimal coverage

---

Key Research Insights: Hardware Layer

1. The Memory Hierarchy is Critical

Modern solutions treat memory as a 4-tier hierarchy:

1. GPU HBM (high bandwidth, tiny capacity) → Attention computation

2. GPU DRAM (medium bandwidth, small capacity) → Active KV cache

3. CPU DRAM (lower bandwidth, larger capacity) → Inactive weights/cache

4. SSD/Flash (low bandwidth, massive capacity) → Long-term storage

Papers like FlexGen and ActiveFlow show that intelligent orchestration across these tiers enables models 10-100x larger than GPU VRAM alone.

2. Hardware Specialization is Emerging

• PIM (L3, PIMphony): Reduces data movement by computing in memory

• HPU: Dedicated units for low-intensity operations

• In-storage computing (InstInfer): Offloads attention to storage controllers

This mirrors the historical shift from general-purpose CPUs to specialized GPUs—we’re now seeing memory-specialized accelerators.

3. Software Must Be Hardware-Aware

Most agent memory papers ignore hardware constraints. This creates a gap:

• Agent designers assume infinite memory (e.g., “store all experiences”)

• Hardware has strict VRAM limits (24-80GB consumer GPUs)

Papers like Combating Memory Walls and DeepSeek-V3 Insights call for hardware-aware agent design.

4. The “Memory Wall” is Multi-Dimensional

Solutions must address multiple dimensions simultaneously.

---

Research Gaps: Hardware × Memory

Critical Missing Areas:

1. Hardware-Aware Agent Architectures

   • Most agent papers (MemGPT, Agentic Memory) don’t specify hardware requirements

   • Need: Agent memory systems that adapt to available hardware

2. Unified Benchmarking

   • No standard for measuring memory performance across hardware tiers

   • Proposed: Memory-throughput-latency Pareto curves for different systems

3. Cross-Layer Optimization

   • Hardware and software optimized independently

   • Need: Co-designed systems (e.g., PIM hardware + agent memory manager)

4. Mobile/Edge Agent Hardware

   • Most work focuses on data center GPUs

   • Gap: Specialized hardware for on-device agents (NPUs, edge accelerators)

5. Energy-Efficient Memory

   • Memory access dominates LLM energy consumption

   • Need: Sustainability metrics in memory system design

---

Practical Implementation Guide: Hardware-Aware Memory Systems

For Developers:

Consumer Hardware (Single GPU: 24GB)

Layer 0: FlexGen-style offloading to CPU
Layer 1: KVQuant (4-bit) + SnapKV (50% pruning)
Layer 2: MemGPT (4K main, vector disk)
Layer 3: LightRAG (local vector DB)

Achieves: ~50K effective context, 10 req/min throughput

Enterprise Hardware (Multi-GPU: 8x80GB)

Layer 0: Jenga (heterogeneous batching)
Layer 1: PagedAttention + SentenceKV
Layer 2: Agentic Memory (trained curation)
Layer 3: MemOS (persistent storage)

Achieves: 1M+ context, 100+ concurrent agents

Mobile/Edge (6GB RAM)

Layer 0: ActiveFlow (DRAM-Flash swapping)
Layer 1: Aggressive quantization (2-bit)
Layer 2: SimpleMem (compressed episodes)
Layer 3: Local-only RAG

Achieves: 3B model, 8K context, <2s latency

---

For Researchers:

High-Impact Research Directions:

1. Hardware-Adaptive Agent Memory

   • Agents that detect available hardware and configure memory systems accordingly

   • Example: Automatically switch between MemGPT (high VRAM) and SimpleMem (low VRAM)

2. PIM + Agent Co-Design

   • Design agent architectures that natively leverage PIM

   • Example: Episodic memory stored in PIM-enabled DIMMs with in-memory search

3. Energy-Aware Memory Management

   • Memory systems that optimize for energy, not just latency/throughput

   • Example: Sleep-like memory consolidation that reduces active DRAM usage

4. Cross-Platform Memory Standards

   • Portable agent memory that works across cloud, edge, and mobile

   • Example: MemOS-style API with hardware abstraction layer

5. Hardware Benchmarking Suite

   • Standardized tests for memory system performance

   • Metrics: tokens/sec, $/token, watts/token, memory efficiency (GB_effective/GB_physical)

---

Key Insights from the Unified Framework: Software/Hardware

1. Complementary Solutions

• KV cache bottlenecks require LLM-level and hardware solutions (quantization, paging)

• Context limits need both agent-level curation (working memory) AND architectural improvements (long-context models)

• Persistence is primarily solved by RAG and agent memory systems

2. Emerging Integration

Recent papers increasingly combine multiple memory types:

• Agentic Memory (2025) = Working Memory + Self-Evolving + RL-enabled

• MemOS (2024) = Infrastructure bridging Agent, LLM, and RAG memory

• Memory-R1 (2025) = Parametric + RL-enabled + Procedural

3. Research Gaps

• Multimodal procedural memory: How do agents learn skills from video/sensory data?

• Hardware-aware agent memory: Most agent papers ignore GPU constraints

• Standardization: No universal API for memory operations (MemOS attempts this)

4. Timeline Evolution

• 2023: Foundation (MemGPT, PagedAttention, Mamba)

• 2024: Specialization (Graph RAG, Multimodal, RL-enabled)

• 2025: Unification (Agentic Memory, SimpleMem, Memory-R1)

---

Practical Implementation Pathways

For Developers Building Agents:

1. Start with Layer 2+3: Implement working memory (MemGPT-style) + RAG persistence

2. Add Layer 4 if needed: Incorporate procedural learning for skill-based tasks

3. Optimize Layer 1 last: KV cache optimization once context becomes bottleneck

For Researchers:

1. Focus on gaps: Multimodal procedural memory, hardware-agent co-design

2. Cross-layer solutions: Systems that address multiple bottlenecks simultaneously

3. Standardization: Contribute to memory OS frameworks

---

Final Integration: Hardware Enables Everything

The hardware layer is foundational but often ignored in agent/LLM memory research. This creates a dangerous disconnect:

• Theoretical capability (agent papers): “Store all experiences indefinitely”

• Practical reality (hardware constraints): 24GB VRAM on consumer GPUs

The most impactful future work will bridge this gap through:

1. Hardware-aware agent design

2. Agent-aware hardware design

3. Unified benchmarking and standards. This unified framework shows that memory bottlenecks require solutions across all layers—from hardware optimization to cognitive architecture design. The most promising recent work (2025) focuses on unified systems that integrate multiple memory types to solve problems holistically rather than in isolation.

The holy grail: An agent memory system that seamlessly scales from mobile phones (6GB) to data centers (terabytes) without algorithmic changes—just like modern operating systems handle memory across diverse hardware.

To this extent, if it hasn’t sunk in yet - Nvidia always wins!!

---