How semantic search over your memory beats keyword search for LLM assistants

When an LLM assistant needs to remember something, the first question is not "what does memory look like on disk" but "how do we find the right memory to hand back." That's a retrieval problem, and the retrieval strategy matters more than the storage layer. Get the retrieval right and a modest storage layer is fine. Get the retrieval wrong and no amount of clever storage will save you.

Gnosem uses semantic search over vector embeddings as its primary retrieval path — specifically cosine similarity on 768-dimension vectors from BAAI/bge-base-en-v1.5, served by Cloudflare Workers AI. This is a deliberate choice against the "just use SQL LIKE" alternative that a lot of memory systems started with. The reasons why are worth walking through carefully, because they're what make cross-vendor memory feel like magic instead of like grep.

The keyword failure mode

Suppose you wrote this memory two weeks ago:

Switched the pricing page from a table layout to a card grid. Users found the columns confusing when comparing plans; the card format with call-to-action buttons per plan converts about 20% better on mobile.

Now you're back on the same project and you ask your assistant:

How did we lay out the checkout screen? I want to keep it consistent.

A keyword search over your memory graph runs something like WHERE content LIKE '%checkout%'. It returns zero matches. The memory you actually wanted — the one about pricing page layout, which is the closest sibling to a checkout screen and shares the design constraints — never surfaces, because the word "checkout" doesn't appear in it.

You could try OR content LIKE '%pricing%'. Now you get the pricing memory. But the assistant has no basis for that OR — the model doesn't know a priori that "pricing" is a keyword to check when the user says "checkout." The keyword approach requires the query to already share vocabulary with the target, and users almost never do.

The same problem shows up everywhere:

• User asks about "auth flow", memory says "login sequence"
• User asks about "database", memory says "postgres"
• User asks about "the bug from Tuesday", memory says "the null pointer in the parser"

Keyword search finds none of these. Semantic search finds all three.

What semantic search actually does

Every memory in Gnosem, on write, is passed through bge-base-en-v1.5 to produce a 768-dimension floating-point vector. That vector is a numerical representation of the memory's meaning — text that talks about similar concepts produces vectors that point in similar directions in 768-dimensional space.

On read, the query goes through the same model, producing a query vector. The retrieval engine (Cloudflare Vectorize) compares the query vector to every stored memory's vector using cosine similarity — literally the cosine of the angle between them. Vectors pointing in nearly the same direction have similarity near 1.0. Vectors pointing in opposite directions have similarity near -1.0. The top-K matches are returned.

Because "checkout screen" and "pricing page layout" both talk about UI layout and conversion — the underlying concepts are close — their vectors cluster in the same region. Cosine similarity picks that up. The keyword mismatch is irrelevant because the model already learned that these words live near each other during pretraining.

The BGE-base-en-v1.5 tradeoff

Gnosem embeds with BGE-base-en-v1.5. There are objectively better English embedding models — larger BGE variants (bge-large, 1024d) and OpenAI's text-embedding-3-large (up to 3072d). They score higher on MTEB, which is the standard embedding benchmark. So why the smaller one?

Three concrete reasons:

1. It's available on Workers AI, at the edge. An embedding call adds a few milliseconds because it runs in the same colo as the Worker. Calling OpenAI's API from Cloudflare adds a round trip to us-east-1 or wherever OpenAI's edge is, easily 100–200ms. When every write triggers an embedding call, that latency compounds.
2. 768 dimensions is small enough to be cheap to store and fast to search. 3072-dim vectors take 4x the storage and roughly 4x the compute per similarity comparison. For a memory store aimed at individual users, the storage and query economics of the smaller model win comfortably over the accuracy gap.
3. It's genuinely competitive on retrieval quality for the kind of content memories contain. Where the larger models pull ahead is niche technical retrieval, code search, and very long documents. General-purpose personal memory — mostly short prose about projects, decisions, and preferences — sits well inside the range where BGE-base is functionally interchangeable with the giants.

If Gnosem were a document search engine over academic papers we'd have made a different call. For personal memory, the tradeoff clearly favors the smaller model.

Filter-then-search: metadata as a first-class citizen

Pure vector search has a scaling problem. If you have 10,000 memories and you cosine-compare all of them on every query, you're doing 10,000 dot products of 768-dim vectors per search. That's not slow, but it's wasteful when 9,900 of those memories belong to another user.

Vectorize supports metadata filters that apply before the similarity search runs. Gnosem uses one on every query:

await env.VECTORIZE.query(queryVec, {
  topK: 10,
  filter: { user_id: currentUserId },
  returnValues: false,
});

Filter-first is not just a performance win — it's how per-user isolation is enforced at the query engine layer. See Building an MCP Server on Cloudflare Workers for the two-layer isolation model. The filter reduces the candidate pool to just this user's memories, then similarity ranks within that pool. Same accuracy, dramatically less compute per query.

You could filter on other metadata too — by tag, by written_by, by date. Gnosem's memory_search today filters on user_id only and does full-population semantic search within that. If future workloads justify it, we'd add explicit tag/date filtering to the search API, but the current pattern is that the semantic ranking is good enough that additional filtering usually adds more friction than clarity.

Re-ranking: sometimes worth it, often not

A common escalation: run vector search to get top-N candidates, then run a heavier cross-encoder model to re-rank those candidates and return the top-K. Cross-encoders read the query and each candidate together, which is more expensive than bi-encoder cosine but produces sharper rankings.

Gnosem does not re-rank. The decision: for personal memory sizes (thousands to low tens of thousands of memories per user), the top-10 from cosine search is already accurate enough that the model consuming it can ignore the ordering and just read all 10. Re-ranking adds a second model call and 100–500ms of latency to shave a few percentage points off ranking quality that the LLM downstream would have ignored anyway.

Where re-ranking earns its keep: massive document corpora (millions of docs), tight top-1 requirements, or precision-sensitive retrieval where the LLM sees only the top result. None of those describe personal memory today. If Gnosem ever grows to team-scoped memory with hundreds of contributors, re-ranking becomes plausibly worth it.

Hybrid search: BM25 + vectors

Hybrid retrieval combines keyword scoring (typically BM25) with vector similarity. The intuition: keyword search is superb at exact matches for IDs, error codes, function names, proper nouns — the cases where the exact string matters and semantic similarity might miss it. Vector search is better at concepts. Combine both and you cover both regimes.

Gnosem does not currently do hybrid search. It's a plausible future addition. The pragmatic reason we haven't: for the kind of natural-language questions LLMs ask (which is the reading side of Gnosem), pure vector search recall has been solidly good enough. The failures we see aren't "we missed a specific error code" — they're rarer and usually about too-recent memories that hadn't quite absorbed into the assistant's context yet.

Cursor and Claude Code do sometimes write memories containing exact identifiers (SHA hashes, error codes, package versions). Those are cases where BM25 would help. It's on the list, but low priority against features like team scope and JSON export.

What gets embedded, and when

One subtle design question: what text goes into the embedding? Gnosem embeds the raw content of the memory, even when the memory is also compressed into structured facts for token-efficient reading (see Memories that Cost Fewer Tokens to Read).

The reason: semantic search should respect natural phrasing. If you write "the auth refactor for the FastAPI backend" and later ask "how did we do login on the Python API," the query and memory should collide in embedding space through the natural-language paths. Embedding the structured form TOPIC=auth | STACK=Python,FastAPI | ACTION=refactor instead of the prose would collapse that similarity by losing the vocabulary the query is likely to use.

Trade-off: this means the compressed form isn't what's being searched. The retrieval ranks memories by how well their raw text matches your query, then returns the compressed form for the reading LLM to consume. Best of both worlds — natural query-matching, token-efficient reading.

The cost model of vector search at Gnosem's scale

Every memory write triggers one Workers AI embedding call. Every search triggers one embedding call for the query, plus one Vectorize query. On Cloudflare's pricing, none of these are individually expensive — Vectorize charges per stored vector and per query, and Workers AI charges by the neuron. For a user with 5,000 memories, the storage and search cost is a small fraction of a dollar a month.

Where cost would matter is if someone tried to run high-frequency polling ("call memory_search every 10 seconds during a session"). Nothing in the protocol prevents this and no MCP client we've observed actually does it, but it's a rate-limit case we watch. In practice, MCP clients call memory_search at natural conversation boundaries — start of a new task, when the user asks a question that suggests prior context — which is a rate an order of magnitude below what would push cost into a concern.

What vector search still misses

Semantic search is not a solved problem. Failure modes to know about:

• Temporal reasoning. "What was our decision from last Thursday?" is a bad query for semantic search because "last Thursday" has no semantic meaning tied to a date. Gnosem does not do temporal retrieval today; the assistant would have to call memory_list with a cursor to walk time.
• Negation. "Memories about databases that are not Postgres" queried semantically will happily return Postgres memories, because the vector for the query is close to the vector for the target. Vector search doesn't understand NOT.
• Aggregation. "How many memories do I have about auth?" is not a retrieval question, it's a count. The right move is memory_list plus assistant-side filtering, not memory_search.

For those cases, MCP clients call memory_list and do the reasoning themselves. Semantic search handles the 90% case cleanly; the other 10% is what the other tools are for.

Try it

The single best demo: write two or three memories in your natural voice about a project, then ask your assistant a question using totally different words — synonyms, aliases, higher-level concepts. Vector search should find your memories anyway. If it does, that's the property you couldn't have gotten from SQL LIKE, and it's the property that makes cross-vendor memory across Claude, ChatGPT, Cursor, and Windsurf feel like a single continuous conversation instead of a set of grep queries. Try it at gnosem.dev.