Cache planner and trajectory rewriter
Caveman includes two independent optimization systems:
- cache planner adds provider-native prompt-cache controls where economics and request shape support them;
- trajectory rewriter asks a model to shorten selected historical agent context and accepts output only after structural checks.
Neither system caches model responses.
Prompt-cache planner
Cache planner inspects a request, identifies stable prefix boundaries, and adds provider-native cache hints. Planning and request optimization make no network calls.
Built-in planners cover Anthropic, OpenAI, Amazon Bedrock, and Google Gemini. Unknown providers pass through unchanged.
Fail-safe cases
Planner keeps original request when it sees:
- record mode;
- unsupported billing tier;
- malformed or ambiguous JSON;
- duplicate JSON keys;
- unsupported provider behavior;
- volatile content at a candidate boundary;
- provider semantics that have drifted from registered capability data.
Provider-specific parsers own exact request shape. Generic planning uses expected call count and provider rate units. It does not invent dollar savings when provider pricing evidence is absent.
Provider observations
A cache hint does not prove a cache hit; provider response usage determines whether a cache read or write occurred. Local planning records remain inferred, and provider observations retain their own evidence basis.
Trajectory rewriter
Trajectory rewriting targets older, lower-value zones in an agent transcript. It currently supports configured Anthropic and OpenAI rewrite models.
Rewrite path requires:
- input exceeds configured threshold before any provider call;
- model and credential are explicitly configured;
- response passes structural acceptance checks;
- failure signals and counts remain present along with exit codes and references;
- output beats configured size threshold;
- exact original is stored behind a recovery pointer.
Rejected rewrites are never inserted into agent context. Provider usage from a rejected attempt is still reported because spend occurred.
Why failures are preserved
Past failures often explain why an agent chose its current approach. Removing an exit code, error class, or failed command can make a later step appear unmotivated. Acceptance therefore checks more than output length.
Recovery and trust
Accepted rewrite is model-generated text, not a lossless encoding. CCR provides access to original trajectory, but callers still need task evaluation before claiming equal quality.
Choosing between systems
| Need | Use |
|---|---|
| Reuse an exact stable provider prefix | Cache planner |
| Shorten old conversational history | Trajectory rewriter |
| Avoid any model-visible byte change | Record mode; cache hints only where request contract treats them as metadata |
| Work offline | Cache planner or Engine; not trajectory rewriter |
Systems can coexist, but evidence must remain separate. Cache usage, Engine token estimates, rewrite-model spend, and downstream model quality are distinct measurements.
Benchmark boundary
Repository cache corpus tests measure safety gates and planner behavior. They do not establish a universal provider savings rate or market ranking. Provider features and prices change; recheck capability data and public documentation before publishing current claims.