How Gnosem's memory_write_bulk dedupes within a single batch

Bulk memory writes are the escape hatch for anyone moving from another memory service (mem0, ChatGPT memory dump, notes app) into Gnosem. The pattern is obvious: pass an array of up to 50 memories to memory_write_bulk, get back an ordered array of results (one per input). Under the hood it parallelizes embeddings and batches the D1 inserts.

What's not obvious is what happens when the batch itself has duplicates.

The obvious approach (that doesn't work)

Gnosem does content-hash dedup on every write: SHA-256 the raw content, look up existing rows with the same user_id + hash, short-circuit if found. Fast, cheap, catches byte-identical repeats.

That works fine for one-at-a-time writes. In a batch, it hits a subtle race:

batch = [
  { content: "I prefer Postgres for greenfield work" },
  { content: "I prefer Postgres for greenfield work" },  // exact dupe
  { content: "I prefer Postgres for greenfield work" },  // also exact dupe
]

The batch handler runs all three entries through embed + optimize in parallel via Promise.allSettled. Each entry independently does the hash lookup. At the moment of lookup, NONE of them are in the database yet — the batch hasn't inserted anything. All three lookups return "not found." All three then proceed to insert. You end up with three identical rows.

The transactional fix — a SERIALIZABLE transaction with per-row locks — isn't available in D1. Even if it were, serializing the batch defeats the whole point of parallel embed calls.

The two-phase reservation

Instead of relying on the storage layer to enforce uniqueness, the batch handler keeps its own in-memory table for the duration of the request. Two phases:

Phase 1: hash + reserve. Walk the input array once. For each entry, compute the SHA-256 hash. Look up in D1 for existing rows with that hash (belt-and-suspenders — catches dupes across prior writes). If found, mark the entry as exact_duplicate: true and remember the existing id. If not found, check the local reservation table:

• If the hash is already reserved by an earlier entry in this batch, add this entry's index to that hash's "waiters" list.
• If the hash is new, reserve it under this entry's index and mark this entry as the "primary" for the hash.

After phase 1, the reservation table looks something like:

{
  "sha256:abc123...": { primary: 0, waiters: [1, 2] },
  "sha256:def456...": { primary: 3, waiters: [] },
}

Entries at indices 1 and 2 will NOT hit the embedding or write path — they're waiters. They inherit whatever the primary (index 0) resolves to.

Phase 2: run primaries, back-fill waiters. Fire embed + optimize + insert for every primary index in parallel via Promise.allSettled. When each primary resolves, back-fill the results array at all the waiter indices with { id: primary.id, created_at: primary.created_at, exact_duplicate: true }.

The result: three identical entries in a batch produce exactly one embedded write, and the response is:

[
  { id: "2cccc06e-4ce9-4bf0-a62d-5be01bec8127", created_at: 1785444857131 },
  { id: "2cccc06e-4ce9-4bf0-a62d-5be01bec8127", created_at: 1785444857131, exact_duplicate: true },
  { id: "2cccc06e-4ce9-4bf0-a62d-5be01bec8127", created_at: 1785444857131, exact_duplicate: true },
]

Order preserved, all three point to the same id, one embedding call fired instead of three.

The tricky edge cases

Two things broke the first cut of this implementation and shipped as follow-up fixes:

The primary took a non-write path. If the primary's own semantic-dedup check (cosine ≥ 0.85 against a prior memory) fired, or if the free-tier limit was hit, the primary would return without an id — but the waiters were already queued expecting an id from the primary. The fix: extract a backfillWaiters(primaryResult) helper and call it on every terminal branch of the primary's write path — success, semantic dedup, hash dedup, free-tier reject, whatever. Every branch produces a result; waiters inherit whatever that result is.

Vectorize is eventually consistent. The semantic dedup check (cosine ≥ 0.85) queries Vectorize. Two identical entries in the same batch will not see each other in Vectorize during phase 1, because neither has been indexed yet. So semantic dedup between in-batch entries is best-effort — content-hash dedup catches identical-string dupes but "paraphrases in the same batch" won't dedup until the second write happens after the first has been indexed (a few seconds). We accept this tradeoff; it's rare in practice and the alternative (serialize the batch) is worse.

Why 50?

The 50-entry-per-batch cap is arbitrary but deliberate. Bigger batches would work — the parallel-embed pattern scales. The cap is there to bound worst-case CPU on a single Worker invocation (50 embeddings × ~30ms each + one D1 batch insert ≈ 1.5s). Above that, callers should chunk their own import into 50-entry pages. We may raise this once we see real usage patterns.

Bulk write is the API used by importers, migration scripts, and any tool moving a corpus of notes into Gnosem in one shot. If you're building one, you can rely on the intra-batch dedup — sending 50 slightly-varying entries that share content with prior writes will produce zero surprise duplicates.