We built a complete caching pipeline for agentic applications. Current ReAct agents spend significant compute reasoning for every query they receive. Previous attempts at addressing this issue include Semantic Caching (GPTCache) and Plan Caching (Agentic Plan Caching). While semantic caching completely fails to generalize across variable changes, Agentic Plan Caching (APC) solves this by retrieving an abstracted plan template and passing it to a lightweight LLM for variable adaptation. However, this LLM-based adaptation phase introduces a significant latency bottleneck, typically taking several hundred milliseconds to over a second per cache hit.
FlashAgent solves this bottleneck by entirely removing the generative LLM from the adaptation step.
Instead of relying on a language model to adapt cached plans, FlashAgent treats agentic cache retrieval as a deterministic parsing problem:
- Ultra-Fast Variable Extraction: We use GLiNER (Generalist Model for Named Entity Recognition) to instantly extract distinct entities and variables from the incoming user query.
- Vector-Based Template Retrieval: The core semantic intent of the query is embedded and matched against a vector database of successful, abstracted plan templates (blueprints).
- Deterministic Prompt Construction The variables extracted by GLiNER are deterministically appended to the agent prompt after the generic plan. This allows optimal KV Cache usage by SGLang as it results in late divergence.
Hit rate + savings across dataset of queries with identical intent but different variables
Latency to produce reasoning of no cache vs cache hit in FlashAgent
Accuracy across complex FinanceBench Queries
Cumulative Impact of FlashAgent across FinanceBench Queries
By replacing the LLM decoder ring with deterministic logic, FlashAgent operates on the extreme edge of the efficiency-accuracy Pareto frontier for agent applications.
- ~100ms Cache Hit Latency: A 10x to 20x speedup compared to LLM-based template adaptation frameworks.
- Near-Zero Marginal Cost: Completely eliminates token generation costs during a cache hit. Vector lookups and GLiNER inference are computationally trivial.
- Complex Question Accuracy Plan based caching enables our 8B param local model to accurately answer complex data dependent tasks within seconds.
The Trade-Off: This architecture is designed for high-volume, structured agentic workflows. Because the variable plugging is deterministic, it sacrifices some of the semantic flexibility of an LLM. It favors raw speed and cost-efficiency over the ability to dynamically adapt a template to mismatched variable counts or complex edge cases.
1. Prerequisites
- Python 3.10+
- SGLang inference engine running your model of choice.
- Google Cloud Vertex AI configured (required for embeddings/agent fallback).
2. Installation Clone the repository and install the required dependencies:
git clone [https://github.com/yourusername/FlashAgent.git](https://github.com/Dantun1/FlashAgent.git)
cd FlashAgent
pip install -r requirements.txt3. Environment Setup
Create a .env file in the root directory and add your Google Cloud Project ID for Vertex AI authentication:
PROJECT_ID="hephaestus-488415"Note: Ensure your local environment is authenticated with Google Cloud by running gcloud auth application-default login.
4. Start Your Inference Engine
FlashAgent defaults to expecting an OpenAI-compatible server running on port 30000. Start your SGLang server with your chosen model (e.g., Llama 3.1 8B FP8 - what we used):
python -m sglang.launch_server --model-path neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8 --port 300005. Basic Usage Example
We have kept test scripts in the testscripts directory, you should be able to run these to see just how effective it is.
(Note: Cache hit/miss metrics and token usage are automatically logged to kv_tracking.csv and the timestamped .log files in your directory.)