Memories that Cost Fewer Tokens to Read

Persistent memory for AI assistants has a token problem. Every time an assistant recalls a memory, that memory has to be pulled back into its context window. A 2,000-character prose memory costs roughly 500 tokens on ingest. Multiply by the ten memories a semantic search returns, and you have spent 5,000 tokens before the model has generated a single word of response. In a 200,000-token context that is fine. In an 8,000-token context, it is a quarter of your budget spent on remembering.

The obvious fix — "just make the memories shorter" — pushes the burden onto the user. They write a long memory because a long memory is what the situation deserves. Amputating it at save time loses information.

Gnosem takes a different approach. It stores every long memory twice: once as the raw prose the user wrote, and once as a compressed structured-facts form generated on write by a small language model. The reading LLM gets the compressed form by default — same meaning, fewer tokens. The raw prose stays available for humans and for cases where phrasing matters.

The write path

When you call memory_write with content longer than 400 characters, the Worker fires two calls in parallel:

1. Embedding — the full raw content goes to Workers AI's @cf/baai/bge-base-en-v1.5 model to produce a 768-dimension vector for semantic search. The embedding always sees the raw text so search hits match natural phrasing.
2. Compression — the raw content goes to @cf/meta/llama-3.1-8b-instruct-fast with a strict system prompt that produces one line of structured key-value pairs.

The system prompt is the load-bearing piece:

You compress user memories into structured facts for AI consumption.
Output ONE line of pipe-separated key: value pairs. Keys are UPPERCASE
labels drawn from this set when applicable: TOPIC, PROJECT, DECISION,
PREFERENCE, PERSON, PLACE, DATE, STACK, PROBLEM, SOLUTION, GOAL,
CONSTRAINT, FACT, EVENT. Values are terse — no filler, no articles
when droppable. Preserve every distinct fact from the input. Do not
add information not in the input. Do not add preamble, quotes, or
explanation. Output ONLY the pipe-separated line.

A real example. Raw input:

I have been working on a Rust web scraper for the past three weeks that captures pricing data from Amazon and eBay for competitive analysis. The stack is Rust with the chromiumoxide crate driving headless Chrome instances. Rate limiting has been the hardest part of the whole project — Amazon aggressively blocks IPs after about 30 requests per second so I have to route through a residential proxy pool from Bright Data which costs around 15 dollars per gigabyte of traffic.

Compressed output:

TOPIC=web_scraper | PROJECT=Rust_scraper | DECISION=use_chromiumoxide
| STACK=Rust,chromiumoxide | PROBLEM=rate_limiting
| CONSTRAINT=Amazon_blocks_after_30_rps
| SOLUTION=residential_proxy_pool_Bright_Data
| FACT=$15_per_GB

Raw: 475 characters. Compressed: 309 characters. 35% fewer bytes, all the same facts, and — critically — the structured form is easier for the next LLM to parse into its own working memory than freeform prose.

The guard: fail-open, no lossy fallback

Compression is not always a win. Content full of proper nouns, IDs, or already-terse writing sometimes comes out longer in the structured form. Gnosem checks this at the byte level after the model responds: if the compressed output is not strictly shorter than the raw, it discards the compression and stores raw only. The memory still saves — the "compression" step is best-effort.

The same applies to model errors. Workers AI can time out. It can rate-limit. It can return an empty string. In every failure case the memory still lands in D1 and Vectorize. The content_optimized column is nullable by design. Fail-open: the write never fails because compression failed.

The read path

By default, memory_search and memory_list return the compressed form as content and the raw prose as content_raw. Any MCP client reading a memory gets:

{
  "id": "…",
  "content": "TOPIC=web_scraper | PROJECT=Rust_scraper | …",
  "content_raw": "I have been working on a Rust web scraper for the past three weeks…",
  "optimized": true,
  "tags": ["project", "scraper"],
  "written_by": "claude-code",
  "created_at": 1785372156851,
  "score": 0.87
}

The reading model consumes the shorter content by default. If your use case needs the original phrasing — quoting the user's own words in a reply, for example — pass raw: true on the search and content becomes the raw prose (and content_raw is dropped).

Short memories (under 400 characters) skip the compression entirely. There is nothing to gain — the structured form would be roughly the same size. Those rows have optimized: false and only a content field.

The cost economics

Every long-memory write incurs one extra Workers AI call — llama-3.1-8b-instruct-fast, priced at fractions of a cent per compression. In exchange, every subsequent read pays back tokens on the client side. If a memory is written once and read ten times over its lifetime, the compression pays for itself many times over. If it is never read again, the compression cost is a rounding error against the storage cost.

The pattern is deliberately asymmetric: pay a small server-side cost on write to save a much larger client-side cost on read, over and over. This is the same trade-off databases make when they build an index at insert time to speed up every subsequent query.

What gets lost — and why we accept it

Structured extraction is lossy for tone. "I have been working on a Rust web scraper" and "PROJECT=Rust_scraper" carry the same information but different vibes. If your assistant is trying to write a response in your voice, the raw form matters. That is why content_raw is preserved verbatim, and why raw: true exists as an escape hatch.

Structured extraction is also nondeterministic in surface form. The same raw memory compressed twice may come back with slightly different key labels — TOPIC=web_scraper vs. SUBJECT=web_scraper. Semantic search is unaffected because it embeds the raw content, not the compressed form. But if you were parsing the compressed output with a strict regex, you would have a bad time. Treat the compressed form as machine-readable prose, not as JSON.

Why this only works because of MCP

If Gnosem had to speak a different protocol to each vendor, this compression would be much harder to justify. Every client's read format would need its own handling. Because MCP is a standard tool-call interface, the compressed form flows uniformly to Claude, ChatGPT, Cursor, Windsurf, Zed, Kimi, and every other MCP client. The server compresses once; every client benefits.

Try it: sign up at gnosem.dev, write a paragraph-length memory, and inspect the response. You will see the optimized: true flag and the compression ratio inline.