The creation pipeline was working:

Content
   ↓
Semantic text
   ↓
Embedding
   ↓
MongoDB

And I had started designing the search system around both semantic and lexical retrieval.

At that point, I thought the next challenge would simply be building the search functionality.

But before search could become reliable, I had to answer a more fundamental question:

What happens when the content that was embedded changes?

That question led me into a completely different part of system design.

I had to think about updates, stale embeddings, partial updates, authorization, versioning, race conditions, and atomic database operations.

The interesting part was that none of these problems existed in the same way during creation.

The Problem With Updating Embedded Content

Suppose BrainZen stores:

Title:
React Performance

Description:
Reducing unnecessary renders

Tags:
react, frontend

and generates:

Embedding:
E1

The relationship is:

Content State 1
      ↕
     E1

Now imagine the user changes the title:

React Performance
        ↓
React Performance Optimization

The content has changed.

But if the database still contains:

Embedding = E1

then we have:

New Content
      ↕
   Old E1

The database technically contains valid data, but the semantic representation is now stale.

That means a future semantic search could retrieve the document based on an embedding that represents the previous version of the content.

So updating content isn't simply:

change a field

It becomes:

change content
        +
possibly regenerate embedding
        +
keep both states consistent

That was the first major realization.

Not Every Update Requires a New Embedding

The next question was:

Does every update require generating a new embedding?

No.

BrainZen has different kinds of fields.

For example:

title
description
tags

contribute to the semantic representation.

But:

link
type

can serve different purposes.

So I separated updates into two categories.

Semantic changes

title
description
tags

If these change:

new content
    ↓
new semantic representation
    ↓
new embedding

Non-semantic changes

For example:

link

If only the link changes:

new content
    ↓
same semantic meaning
    ↓
existing embedding can remain

This gave me an important rule:

The need for a new embedding should be determined by changes to the semantic representation, not by the fact that an update happened.

Partial Updates Make This More Interesting

BrainZen uses a PATCH-style update.

That means the client doesn't necessarily send the entire document.

For example:

{
  "title": "React Performance Optimization",
  "version": 5
}

This means:

Change the title, but keep everything else.

So the backend can't simply look at the request and assume that the request represents the complete new document.

It has to construct the final state.

Conceptually:

Existing document
        +
PATCH fields
        ↓
Final document state

For example:

Existing:
title       = React Performance
description = Reducing renders
tags        = react, frontend

PATCH:
title       = React Performance Optimization

Final:
title       = React Performance Optimization
description = Reducing renders
tags        = react, frontend

Only after constructing this final state can the backend determine:

Did the semantic representation actually change?

This was an important distinction for me.

The request isn't necessarily the new document.

It is an instruction for how to construct the new document.

Tags Made Updates More Complicated

Tags already had two representations during creation.

The frontend works with:

["react", "frontend"]

while MongoDB stores tag references as ObjectIds.

That becomes important during updates.

Suppose the database contains:

ObjectId("A")
ObjectId("B")

but the frontend sends:

["react", "frontend"]

The backend has to resolve those names again before comparing or persisting them.

And when determining whether tags changed, order shouldn't matter.

These should represent the same tag set:

["react", "frontend"]

and:

["frontend", "react"]

So I normalize the values before comparing them.

The larger lesson was:

Before comparing structured data, I need to make sure I'm comparing the same representation.

Otherwise I can accidentally detect a change that isn't really a change.

The Backend Can't Simply Trust the Frontend Embedding

Because BrainZen currently generates embeddings in the browser, another question appeared:

How does the backend know that the embedding actually corresponds to the new content?

The answer is: it can't completely prove that with the current architecture.

The backend can verify that:

embedding exists
        ↓
embedding has 768 dimensions

But it isn't independently running the embedding model to verify:

title + description + tags
        ↓
this exact embedding

That would require generating the embedding on the backend as well.

So I separated the responsibilities.

The frontend handles:

semantic text
     ↓
embedding generation

The backend handles:

validation
authorization
consistency rules
version protection
persistence

This made the boundary much clearer.

Then I Ran Into Concurrency

This was probably the most important part of the update design.

Imagine the user changes a document twice very quickly.

We can have:

Update A
old content → generate E2

Update B
old content → generate E3

Both requests started from:

version = 5

Now imagine:

E3 finishes first
    ↓
saved successfully

E2 finishes later
    ↓
saved successfully

Without some protection, the older operation could overwrite the newer one.

We could end up with something like:

Content = state from B
Embedding = state from A

That violates the relationship we were trying to protect.

This is where I learned about optimistic concurrency control.

Versioning as a Concurrency Guard

Instead of assuming that an update is still working with the latest document, the client sends the version it originally read.

For example:

Frontend has:
version = 5

It sends:

{
  "title": "New Title",
  "version": 5
}

The backend then performs an update only if the database still contains:

version = 5

Conceptually:

Update request
      ↓
"Update this document IF version = 5"
      ↓
Database

If the document is still version 5:

5 → 6

The update succeeds.

But if another request already changed it:

Database:
version = 6

then the condition doesn't match.

The update doesn't happen.

The backend can then return:

409 Conflict

The important thing here is that the application doesn't have to lock the document while someone is editing it.

Instead, it detects that the state became stale when the update is committed.

That's why this is called optimistic concurrency.

Why $inc Matters

MongoDB gives us an atomic increment operation:

$inc: {
    version: 1
}

If the document is currently:

version = 5

MongoDB changes it to:

version = 6

as part of the update.

So the update can conceptually be:

Find:
    _id = X
    userId = Y
    version = 5

Then:
    set new fields
    increment version

This is much safer than treating:

read version
calculate version + 1
write version

as separate application-level operations.

Atomic Update: Another Important Lesson

I also learned that updating related pieces of state should be treated as one logical operation.

For a content update, I don't want:

Update title
     ↓
separate operation
     ↓
Update embedding
     ↓
separate operation
     ↓
Update version

because a failure between those operations could leave the document in an inconsistent state.

Instead, MongoDB can perform the changes together:

$set
  title
  description
  tags
  embedding

$inc
  version

as one update operation.

The desired invariant becomes:

Version N
    ↕
Content state N
    ↕
Embedding representing state N

That's a much stronger mental model than simply thinking:

"I have a content document with an embedding field."

What About Delete?

Delete turned out to have a similar consistency concern.

The embedding itself doesn't need a separate delete operation because it is stored inside the content document.

So:

Delete content
      ↓
embedding disappears with it

But BrainZen also has related share/link documents.

So deleting content can involve:

Content
   ↓
related share

I therefore treated the deletion of the content and its related share cleanup as one database transaction.

The goal is:

Delete content
+
Delete related share
        ↓
both succeed

or:

something fails
        ↓
rollback

rather than leaving behind a partially cleaned-up state.

The Backend Update Pipeline

After working through all of this, the update flow became much clearer to me.

PATCH /content/:id
        ↓
Authentication
        ↓
Validate request
        ↓
Validate Content ID
        ↓
Find content belonging to user
        ↓
Build final document state
        ↓
Did semantic fields change?
      /        \
    NO          YES
    ↓            ↓
keep          require
embedding     new embedding
      \        /
       ↓
Check expected version
        ↓
Atomic MongoDB update
        ↓
Increment version
        ↓
Return new version

And the key invariant is:

Content state N
      ↕
Embedding for state N
      ↕
Version N

The Bigger Lesson

What started as:

"I need to add semantic search."

has turned into a much broader systems problem.

The embedding itself is actually only one component.

The real system looks more like:

                    Content
                       ↓
              Semantic Representation
                       ↓
                    Embedding
                       ↓
                    Storage
                       ↓
              ┌────────┴────────┐
              ↓                 ↓
        Semantic Search    Lexical Search
              ↓                 ↓
         Vector ranking       BM25
              └────────┬────────┘
                       ↓
                      RRF
                       ↓
                   Filtering
                       ↓
                    Results

But now there's another dimension around the entire system:

Create
  ↓
Update
  ↓
Version
  ↓
Consistency
  ↓
Delete

Search isn't reliable just because retrieval works.

The data being retrieved has to remain correct over time.

That's probably the biggest thing I learned from this part of BrainZen:

Building a search system isn't only about finding relevant information. It's also about making sure the information being searched is represented correctly as the system changes.

And now that BrainZen can reason about content changes, embeddings, and consistency, the next challenge is the part I initially thought would be the main feature:

actually building the retrieval system — lexical search, semantic search, hybrid retrieval, filtering, and ranking.

It turns out that was only the beginning.

Once the embedding pipeline was working, I had to understand what actually happens around that vector: what information should be embedded, how the backend should validate it, how authentication fits into the flow, how tags are represented, and eventually how semantic search will work alongside traditional keyword search.

This made me realize that search isn't a single feature. It's an entire system.

What Actually Gets Embedded?

A BrainZen content document contains several different kinds of information:

Content
├── title
├── description
├── tags
├── type
├── userId
├── link
└── embedding

But not all of these fields have the same purpose.

For semantic search, I decided that the meaningful representation should come from:

title + description + tags

Fields like userId and type serve different purposes. userId is about authorization and data isolation, while type can be used as a structured filter.

This distinction is important because the embedding should represent what the content means, not every piece of metadata attached to it.

So before generating a vector, BrainZen first converts the structured content into semantic text.

For example:

React Performance
Reducing unnecessary renders
react frontend

That text is then passed to the embedding model.

I intentionally separated these responsibilities:

buildEmbeddingText()
        ↓
semantic text
        ↓
generateEmbedding()
        ↓
vector

The first function answers what represents the content?

The second answers how do I turn that representation into a vector?

The Create Flow Is More Than Model.create()

Once the frontend has the embedding, it sends the content to the backend.

The complete flow looks like this:

Frontend
   ↓
Generate embedding
   ↓
POST /api/v1/content
   ↓
Authentication middleware
   ↓
Zod validation
   ↓
Resolve tags
   ↓
Get authenticated userId
   ↓
Create MongoDB document
   ↓
Return response

One of the important things I learned here is that the backend cannot trust frontend data simply because my own frontend generated it.

The request is still runtime data coming from a client.

So the backend validates it with Zod before continuing.

Conceptually:

req.body
   ↓
untrusted data
   ↓
Zod
   ↓
validated + normalized data
   ↓
business logic
   ↓
database

This also means I should continue using the validated result rather than going back to the original req.body.

Authentication and User Data Are Different Things

Another distinction that became much clearer while implementing the flow was where userId should come from.

The frontend does not send the user's ID.

Instead:

JWT cookie
    ↓
auth middleware
    ↓
verify token
    ↓
extract userId
    ↓
req.userId

The controller then uses that authenticated identity when creating the document.

This gives me an important security boundary:

The client provides the content. The server determines the identity.

If the frontend were allowed to send the userId, a malicious client could simply replace it with another user's ID.

So authentication isn't just about deciding whether someone is logged in. It also establishes who the request belongs to.

Tags Have Two Representations

Tags introduced another interesting transformation.

The frontend naturally works with:

["react", "frontend", "performance"]

But MongoDB stores references to tag documents as ObjectIds.

So the backend performs:

"react"
   ↓
find/create tag
   ↓
ObjectId

for each tag.

The frontend therefore doesn't need to understand MongoDB's internal representation.

It sends:

string[]

and the backend converts that into:

ObjectId[]

before persistence.

This separation keeps the client-facing representation simple while allowing the database to maintain relationships between documents.

Embeddings Have a Contract

Another thing I had to understand is that an embedding isn't simply "an array of numbers."

BrainZen currently uses:

all-mpnet-base-v2
        ↓
768-dimensional vector

That dimension is part of the system's contract.

The document embedding and the future query embedding must be generated using the same embedding model and configuration.

Otherwise, even if both outputs happen to be arrays of 768 numbers, they aren't necessarily comparable in a meaningful way.

So the architecture is really:

Same model
   ├── document → embedding
   └── query    → embedding

Both vectors live in the same embedding space, which is what makes similarity search meaningful.

Why Semantic Search Isn't Enough

Once I understood the embedding pipeline, the next question was:

Should BrainZen use semantic search for everything?

No.

Semantic search is excellent at understanding meaning, but sometimes the exact terminology matters.

Consider:

MongoDB ObjectId

If I'm searching for that specific technical term, I don't necessarily want the system to return something merely because it discusses the broader concept of database identifiers.

This is where lexical search becomes important.

Semantic search asks:

"What content means something similar to this query?"

Lexical search asks:

"What content contains or strongly matches these terms?"

They're solving different problems.

BrainZen's Search Architecture

That led me toward a hybrid architecture:

                 Search Query
                      │
             ┌────────┴────────┐
             ↓                 ↓
      Semantic Search    Lexical Search
             │                 │
             ↓                 ↓
       Vector Ranking       BM25
             │                 │
             └────────┬────────┘
                      ↓
                     RRF
                      ↓
               Final Ranking
                      ↓
             Structured Filters
                      ↓
                  Top Results

The semantic side will generate a query embedding and retrieve semantically similar documents.

The lexical side will use traditional text retrieval and BM25-style ranking.

The two systems produce independently ranked results.

Those rankings can then be combined using Reciprocal Rank Fusion (RRF).

One important thing I learned here is that RRF isn't simply adding the semantic score to the lexical score.

The two systems produce different types of scores, so combining their raw numbers directly isn't necessarily meaningful.

Instead, RRF works from the rank positions of documents in each result list.

Tags Can Play Three Roles

Tags became particularly interesting because they can participate in multiple parts of the architecture.

They can contribute to the semantic representation:

title + description + tags
        ↓
embedding

They can participate in lexical search:

tags
 ↓
keyword retrieval

And they can also be used as a structured filter:

tag = "react"

So the same piece of data can serve three different purposes:

Tags
├── Semantic meaning
├── Lexical matching
└── Structured filtering

These aren't interchangeable operations.

They're different ways of using the same underlying information.

The Bigger Lesson

The biggest thing I learned wasn't how to generate a 768-dimensional vector.

It was understanding that search is a pipeline of responsibilities.

Content starts as structured user input:

Content
   ↓
Semantic representation
   ↓
Embedding
   ↓
MongoDB
   ↓
Retrieval
   ↓
Ranking
   ↓
Filtering
   ↓
Results

And different parts of that pipeline solve different problems.

• Embeddings represent semantic meaning.

• Vector search retrieves semantically similar candidates.

• Lexical search handles terminology and keyword-based retrieval.

• BM25 ranks lexical matches.

• RRF combines independently ranked retrieval systems.

• Structured filters enforce constraints such as user ownership, type, or tags.

• Top-K controls how many results are returned.

What initially looked like "adding AI search" is turning into something much more interesting: designing an actual retrieval system.

And now that the creation pipeline and embedding foundation are in place, the next problem becomes even more important:

What happens when the user edits the content?

Because if the content changes but its embedding doesn't, BrainZen's search system can become inconsistent.

That's where I'm heading next.

I was wrong.

The very first hurdle was a much more foundational question: Where exactly is this embedding model going to live? That single decision dictates your hosting costs, security boundaries, performance bottlenecks, and the entire shape of your infrastructure.

The Current State of BrainZen

At a high level, BrainZen is split into three main pieces. The crucial detail here is that both the documents you save and the queries you run have to pass through the exact same embedding model. If they come from different embedding spaces, comparing them is mathematically meaningless.

Plaintext

Frontend (React + TypeScript)
  │
  │ generates embeddings
  ↓
Embedding Model (all-mpnet-base-v2)
  │
  │ 768-dimensional vector
  ↓
Backend (Express + TypeScript)
  │
  ↓
MongoDB Atlas
  ├── Content
  ├── Tags
  └── Embeddings

Where Should the Model Run?

I had a few distinct architectural paths I could take, each with its own baggage.

External Embedding API The easiest route. Send the text from the backend to an external API, get a vector back, and save it to Mongo. But this introduces a third-party dependency, API quotas, latency, and usage costs. I didn't want BrainZen's public deployment tethered to my personal API keys.

Run it in the Backend This keeps the data in-house. But now my lightweight Vercel backend has to lug around an ML model and a Python-esque runtime. For my current deployment setup, that's immediate technical debt.

Separate Inference Service A dedicated microservice just for handling models. It’s the "enterprise" answer, but it means managing another server, another CI/CD pipeline, and paying for dedicated compute. Overkill.

The Browser (The Winner) I ultimately decided to run the model directly on the client.

By using Transformers.js with the Xenova/all-mpnet-base-v2 model, I can generate embeddings right inside the user's browser. It completely eliminates external APIs, API keys, and dedicated ML servers.

The obvious tradeoff? The user’s browser has to download the model and spend CPU cycles running the inference. It’s not the "perfect" architecture for every app, but it was exactly the right compromise for BrainZen's current constraints.

Solving the Model-Loading Problem

Once you push ML to the browser, you immediately hit a performance wall. You absolutely cannot do this:

• Generate embedding 1 → Load the model

• Generate embedding 2 → Load the model again

The model needs to be loaded into memory exactly once and reused. I didn't need a heavy Singleton class to fix this; a simple module-level cache handles it perfectly:

TypeScript

import { pipeline } from "@huggingface/transformers";

let extractor: any = null;

async function getExtractor() {
  if (!extractor) {
    extractor = await pipeline(
      "feature-extraction",
      "Xenova/all-mpnet-base-v2"
    );
  }
  return extractor;
}

The first time getExtractor() is called, it downloads and caches the model pipeline. Every subsequent keystroke or document upload just reuses that existing instance in memory.

Turning Text into Vectors

The actual function doing the heavy lifting is surprisingly tiny:

TypeScript

export async function generateEmbedding(text: string): Promise<number[]> {
  const model = await getExtractor();

  const output = await model(text, {
    pooling: "mean",
    normalize: true
  });

  return output.tolist()[0];
}

Two parameters do the heavy lifting here. Setting pooling: "mean" condenses the model's token-level gibberish into a single, usable representation of the text. Setting normalize: true prepares the resulting vector for the cosine-similarity math I use later on.

Finally, output.tolist()[0] converts the tensor into a standard JavaScript array of 768 floating-point numbers. That array gets shipped off to the database.

Why Stick with MongoDB?

Since I was already using MongoDB as BrainZen's source of truth, standing up a dedicated vector database (like Pinecone or Milvus) felt entirely unnecessary.

I just appended an embedding array to my existing schema:

Plaintext

Content
 ├── title
 ├── description
 ├── tags
 ├── type
 ├── userId
 └── embedding  <-- 768 numbers live here

MongoDB Atlas Vector Search hooks right into this field, letting me run semantic queries natively alongside my standard document fetches.

Why Semantic Search Isn't Enough

Here is the most practical lesson I learned during this build: Semantic search cannot fully replace lexical (keyword) search.

If I search for MongoDB ObjectId, I am looking for that exact, specific technical syntax. A purely semantic model might try to be "helpful" and return documents about abstract database identifiers because they mean the same thing conceptually. That's a terrible user experience.

Semantic search is for meaning. Lexical search is for exact terms.

BrainZen's eventual pipeline uses Hybrid Search, funneling both semantic and lexical results through a Reciprocal Rank Fusion (RRF) algorithm to score the best possible matches before applying standard filters.

The Real Takeaway

The fun part of this wasn't actually writing the generateEmbedding() function. It was figuring out where that computation actually belonged.

Choosing a model isn't just about picking the one with the best benchmark scores. You are inherently choosing your deployment strategy, your infrastructure costs, and how the rest of your system has to communicate.

Instead of just npm install-ing a search library and calling it a day, building BrainZen this way forced me to actually understand why the architecture works in the first place.

But once I started trying to make BrainZen—my second-brain app—actually understand what I'm looking for by meaning while still catching exact keywords, tags, and names, I realized real search is a whole different beast. Over the past few days, I dove deep into how modern search systems work and started architecting the search layer for BrainZen. Here's a breakdown of what I ran into and what I learned.

1. Semantic Search (Or: Searching by Meaning)

Traditional keyword search basically asks one rigid question: "Does this document contain the exact words I typed?"

Semantic search, on the other hand, asks: "Is this document actually about the same idea as my query?"

To pull this off, you use embeddings. An embedding model takes a chunk of text and turns it into a high-dimensional vector of floating-point numbers. For example:

Plaintext

"I want to learn how databases organize information" 
↓ (Embedding Model) 
[0.12, -0.84, 0.31, ...]

The individual numbers don't mean much on their own to a human. The magic is that pieces of text with similar meanings get mapped to vectors that sit close to each other in a multi-dimensional vector space. That lets you compare a query and a document based on conceptual closeness rather than spelling.

To do this efficiently at scale, you rely on approximate nearest-neighbor (ANN) algorithms like HNSW (Hierarchical Navigable Small World), which lets you search massive vector spaces quickly without having to do an exhaustive mathematical comparison against every single vector in your database.

2. Why Semantic Search Isn't Enough

This was probably the biggest reality check for me.

Suppose I search BrainZen for MongoDB ObjectId. A purely semantic model might deduce that my query has something to do with database identifiers. But sometimes, I don't want a vaguely related philosophical note on IDs—I want the exact string ObjectId.

That’s where lexical search comes back into play. Lexical search focuses entirely on exact terms, making it indispensable for:

• Technical terms, names, and exact phrases

• Product codes, error messages, and code snippets

• Specific tags and identifiers

In short: semantic and lexical search solve two totally different problems. You really want both.

3. How Lexical Search Works Under the Hood

On the lexical side, everything revolves around the inverted index. Instead of mapping documents to their words, an inverted index flips it around to map terms directly to the documents containing them:

• MongoDB \(\rightarrow\) Document 1, Document 3

• React \(\rightarrow\) Document 2

• ObjectId \(\rightarrow\) Document 3

Building a proper lexical pipeline means dealing with text processing steps like tokenization, normalization, stop-word removal, stemming, and lemmatization, all of which feed into scoring algorithms like BM25. BM25 is the industry workhorse for lexical ranking because it doesn't just check if a word exists—it accounts for term frequency, document length, and how rare or common that term is across your entire collection.

4. Putting Them Together: Hybrid Search & RRF

So, I don't have to choose. I can run both lexical and semantic retrieval in parallel, but that creates a new issue: score mismatch.

A raw vector distance score and a BM25 lexical score speak totally different languages; you can't just add them together mathematically. To solve this, I looked into Reciprocal Rank Fusion (RRF).

Instead of comparing raw scores, RRF looks at the rank position of a document in each result list. If a document pops up near the top of both the lexical search and the semantic search, RRF boosts its final standing. It’s a clean way to combine evidence from multiple systems without overcomplicating the math.

Plaintext

               User Query
                   │
         ┌─────────┴─────────┐
         ↓                   ↓
   Lexical Search      Semantic Search
         │                   │
         └─────────┬─────────┘
                   ↓
                  RRF
                   ↓
             Top-K Results

5. The Catch: Embedding Lifecycle Management

Once I mapped out the pipeline, I realized generating embeddings isn't a "set it and forget it" task.

Imagine a note starts like this:

Title: Learning MongoDB

Description: Understanding indexes and query optimization

The system generates an embedding. But what happens tomorrow when I edit the note to add aggregation pipelines? That original vector is suddenly stale.

Building a real app means handling the whole embedding lifecycle: tracking content updates, detecting stale vectors, generating new ones asynchronously, handling API failures, and cleaning up vector storage when a note gets deleted. Keeping your raw content and your vector indices in sync turns out to be a fun engineering challenge on its own.

Wrapping Up

Working on BrainZen taught me that search isn't a single algorithm—it's an entire pipeline.

From text preprocessing and inverted indexes to vector spaces, RRF fusion, and metadata filtering, every piece handles a specific failure mode of the others. Lexical finds the exact matches, semantic finds the meaning, RRF merges them, and filters make sure you only see what you're actually allowed to look at.

I'm still actively building and refining this in BrainZen. If there's one thing I've learned, it's that building things from scratch forces you to understand why tools like Elasticsearch, Pinecone, or pgvector do what they do under the hood.

That's what Learning with Yahya is all about: build something, break it, figure out why it broke, and write down what happened.