Vector embeddings and semantic chunking, under the hood

Every number and chart on this page comes from running nomic-embed-text-v1.5, an open 768-dimension embedding model, on the examples shown. This note goes deeper into the two steps RAG: what it is and how it works treats as boxes: embed and chunk.

Part 1: Embeddings

From text to one vector, step by step

flowchart TD
T[Text] -->|WordPiece tokenizer| IDS[14 token ids]
IDS -->|embedding lookup| TOK[14 vectors x 768, no context yet]
TOK -->|12 transformer layers| CTX[14 contextual vectors x 768]
CTX -->|mean pooling, then L2 normalize| V[1 embedding: 768 numbers, length 1]

1. Tokenize. The model never sees characters, only ids from a fixed vocabulary of subword pieces. nomic-embed-text uses BERT’s lowercasing WordPiece tokenizer (about 30,000 pieces). Real output:

"search_query: How do I rotate the API keys?"
[CLS] search _ query : how do i rotate the api keys ? [SEP]
 101  3945 1035 23032 1024 2129 2079 1045 24357 1996 17928 6309 1029 102

"Tokenization splits unbelievably long words."
[CLS] token ##ization splits un ##bel ##ie ##va ##bly long words . [SEP]

Common words are one token; rare ones are split into pieces (## means “continues the previous piece”). [CLS] and [SEP] mark the start and end. Two consequences: identifiers like get_with_retry or ERR_4012 become several generic pieces, which is one reason embeddings are weak at exact-match lookups. And the model has a maximum input length in tokens (8,192 for nomic, only 512 for many BERT-based models); anything longer is usually truncated without an error, so the end of an over-long chunk is invisible to search.

2. Look up. Each id indexes a row of a learned matrix, giving one 768-number vector per token. At this point “keys” has the same vector everywhere, whether it’s API keys or piano keys.

3. Contextualize. Twelve transformer layers each let every token attend to every other token and update its vector. After them, the vector for “keys” reflects “API” and “rotate”. This is the part that makes modern embeddings understand paraphrase and word sense.

4. Pool. The model needs one vector for the whole text, so it combines the token vectors. Common choices: mean pooling (average all tokens; nomic, E5, most Sentence-Transformers models), CLS pooling (use the [CLS] token’s vector; BGE), or last-token pooling for embedders built on decoder LLMs.

5. Normalize. Divide by the vector’s length so every embedding sits on the unit sphere. Then the dot product of two embeddings is their cosine similarity:

cosine(a, b) = (a · b) / (‖a‖ ‖b‖)   →   a · b   when ‖a‖ = ‖b‖ = 1

Cosine ranges from −1 to 1, and only the direction matters: a long document and a one-line question can still be close.

How the model learns “similar”

A pretrained language model’s pooled vectors are poor for search out of the box. Embedding models are fine-tuned with contrastive learning on pairs of texts that belong together: a question and the passage that answers it, a title and its abstract, two paraphrases.

flowchart LR
Q[Query: how often do API keys rotate?]
P[Positive: keys are rotated every 90 days]
N1[Negative: reset links expire after 30 minutes]
N2[Hard negative: service accounts do not use API keys]
Q ==>|pull together| P
Q -.->|push apart| N1
Q -.->|push apart| N2

The standard objective is InfoNCE: treat the matching passage as the correct class among a batch of candidates, and minimize cross-entropy over the similarity scores.

                       exp(sim(q, d⁺) / τ)
L = − log  ────────────────────────────────────────────────
            exp(sim(q, d⁺) / τ)  +  Σⱼ exp(sim(q, dⱼ⁻) / τ)

• d⁺ is the true passage; the dⱼ⁻ are negatives. In-batch negatives reuse the other pairs’ passages in the same batch, so large batches give many negatives for free.
• Hard negatives, passages that look relevant but aren’t (like “service accounts don’t use API keys”), teach the fine distinctions that matter most in retrieval. They’re mined with an earlier model or with BM25.
• The temperature τ (often between 0.01 and 0.05) sharpens the softmax so small similarity gaps produce large gradients.

Recipes like DPR, Sentence-BERT, E5, and Nomic Embed combine a large weakly supervised stage (hundreds of millions of web pairs) with a smaller stage on curated pairs with hard negatives. Many models also train with task prefixes (search_query: / search_document: for nomic, query: / passage: for E5), so the same model can embed a short question and a long passage asymmetrically.

What “close” looks like, with real numbers

One question against five candidates, each designed to probe something:

What this shows:

• Paraphrase works. No shared content words besides “battery”, yet it’s the top match.
• Word sense works. The artillery battery is far below the laptop-battery texts even though it shares the key word.
• Negation mostly doesn’t. “Drain faster” scores almost exactly like the paraphrase. Embeddings capture what a text is about far better than what it asserts. Here, retrieving it is harmless; the LLM reading it will notice. But never rely on embeddings alone to distinguish “X is allowed” from “X is not allowed”.
• Prefixes matter. Without them, the negation actually beats the paraphrase (0.860 vs. 0.823). Using the prefixes the model was trained with is the cheapest retrieval improvement there is.
• There’s no natural zero. Unrelated text still scores 0.31.

That last point is general. Embeddings occupy a narrow cone of the space rather than spreading evenly in every direction, a property called anisotropy. Across 15 sentences on three unrelated topics, the average similarity between topics was 0.555 and the lowest was 0.488, while sentences within a topic averaged 0.654. The useful signal is the gap, not the absolute value. That’s why similarity thresholds don’t transfer between models, and why you rank instead of thresholding.

Seeing the space

768 dimensions can’t be drawn, but projecting onto the two directions of greatest variance (PCA) shows the structure. Five sentences each about cooking, databases, and football:

The clusters separate without the model ever being told the topics. Keep the caveat in mind: the two axes explain 18% and 12% of the variance. Most of what distinguishes these vectors isn’t visible in any 2D picture, so use plots like this for intuition, not for judging retrieval quality.

Searching millions of vectors

Retrieval means finding the k vectors with the highest dot product with the query. Doing it exactly (“brute force”) compares against every vector, which is fine into the hundreds of thousands and is what matrix @ q in the RAG note does. Beyond that, vector databases use approximate nearest neighbor (ANN) indexes that trade a little recall for a lot of speed.

HNSW (Hierarchical Navigable Small World), the most common, builds a layered graph. Every vector is on layer 0, linked to its nearest neighbors; a random, exponentially shrinking subset also appears on higher layers with longer links. A search enters at the top, greedily hops toward the query, and drops a layer each time it can’t get closer.

Its knobs: M (links per node: more means better recall and more memory), ef_construction (build-time thoroughness), and ef_search (how many candidates to keep while searching: the main recall-vs-latency dial at query time). Two other families you’ll meet: IVF clusters the vectors with k-means and searches only the clusters nearest the query; product quantization (PQ) compresses each vector into a few bytes by quantizing slices of it separately, trading accuracy for memory. FAISS implements all three.

Memory adds up quickly. For 1 million chunks at 768 dimensions:

Plus the index’s own overhead (HNSW links, IVF assignments). Two common savings: quantize and re-score the top candidates with full-precision vectors, or use a model trained with Matryoshka representation learning (nomic-embed-text-v1.5 is one), whose vectors can be truncated to their first 256 or 128 dimensions with only a modest loss in quality.

Part 2: Chunking

Why chunk at all

Three forces set chunk size:

• The embedding model’s input limit. Anything past it is truncated and unsearchable.
• One vector per chunk. Pooling averages everything in the chunk. A chunk covering three topics gets a vector that is a blur of all three and matches none of them sharply. Smaller, single-topic chunks give sharper vectors.
• The prompt budget. You want several chunks in the prompt, each self-contained enough to be understood without its neighbors.

Too small and a chunk loses the context needed to interpret it (“It expires after 30 minutes.” What does?). Too big and its vector blurs. Most systems land between 200 and 800 tokens, with structure deciding the exact cuts.

The test page

To compare strategies, here’s a 12-sentence wiki page with three topics of four sentences each. The ideal chunks are obvious to a human: deploys, on-call, travel.

[1] Deploys go out from the main branch every weekday at 10am. [2] The pipeline runs unit tests, builds a container
image, and pushes it to the registry. [3] Canary instances take 5% of traffic for thirty minutes before the full
rollout. [4] If error rates rise during the canary, the deploy rolls back automatically.
[5] Each week one engineer is the primary on-call. [6] The on-call engineer carries the pager and must acknowledge
alerts within fifteen minutes. [7] Handover happens on Monday mornings with a short review of open incidents. [8] If
the primary does not respond, the alert escalates to the secondary.
[9] Travel must be booked through the company portal. [10] Economy class is required for flights under six hours.
[11] Hotel costs are reimbursed up to the city rate listed in the policy. [12] Keep all receipts and submit them
within thirty days of returning.

Fixed-size and recursive splitting

Fixed-size chunking slices every N characters or tokens. At 200 characters, the second chunk of this page ends with “…carries the pager and must ack” and the third begins “nowledge alerts within fifteen minutes”. Words and facts get cut in half. Overlap (repeating the last 10 to 20% of each chunk at the start of the next) softens this, at the cost of storing and embedding text twice.

Recursive splitting (LangChain’s RecursiveCharacterTextSplitter) tries separators in order of strength: paragraphs, then lines, then spaces, then characters, and only falls back to a weaker one when a piece is still too big. At 200 characters it produced the same boundaries here but never split a word (“…and must” / “acknowledge alerts…”). Its real strength shows on structured documents, where paragraph and heading breaks line up with topic changes. Structure-aware splitters go further: Markdown splitters cut at headings and keep the heading path as metadata; code splitters cut at function and class boundaries.

Semantic chunking: cut where the meaning jumps

Semantic chunking, popularized by Greg Kamradt’s “5 Levels of Text Splitting”, uses the embedding model itself to find topic boundaries:

flowchart LR
S[Split into sentences] --> W[Window: each sentence plus its neighbours]
W --> E[Embed every window]
E --> D[Cosine distance between consecutive windows]
D --> T[Threshold: e.g. 80th percentile of distances]
T --> C[Cut after every sentence whose distance exceeds it]

The whole algorithm, as run for this page:

import re

import numpy as np


def semantic_chunks(text, embed, buffer=1, percentile=80):
    """Split text where the meaning jumps. `embed` maps a list of strings to a list of vectors."""
    sentences = re.split(r"(?<=[.!?])\s+", text.strip())
    # Embed each sentence together with its neighbours to smooth out noise.
    windows = [" ".join(sentences[max(0, i - buffer) : i + buffer + 1]) for i in range(len(sentences))]
    v = np.array(embed(windows))
    v /= np.linalg.norm(v, axis=1, keepdims=True)
    distances = 1 - np.sum(v[:-1] * v[1:], axis=1)  # cosine distance between neighbours
    threshold = np.percentile(distances, percentile)
    chunks, start = [], 0
    for i, d in enumerate(distances):
        if d > threshold:  # big jump in meaning: start a new chunk after sentence i
            chunks.append(" ".join(sentences[start : i + 1]))
            start = i + 1
    chunks.append(" ".join(sentences[start:]))
    return chunks

Pass any embedding function, for example embed = init_embeddings("<provider>:<embedding-model>").embed_documents, adding the model’s document prefix if it uses one. Here are the 11 distances it computed for the test page, with the default window of one neighbor on each side:

Reading the chart:

• After sentence 8 (on-call → travel) is the tallest peak, 0.231. A clean cut in the right place.
• After sentence 4 (deploys → on-call) is low, 0.072. The peak shows up one gap later, after sentence 5. Windowing explains it: the windows around sentences 4 and 5 share two of their three sentences, so the boundary is smeared across neighboring gaps, and its peak landed one sentence late. Sentence 5, “Each week one engineer is the primary on-call”, ended up as the last sentence of the deploys chunk.
• After sentence 7 reaches 0.1412, exactly the threshold, and doesn’t split because the rule is “greater than”. A slightly different percentile would have added a wrong cut in the middle of the on-call section.

Changing just the window size:

That sensitivity is the main practical lesson. Semantic chunking has no notion of a “right” answer, only a percentile and a window, and small changes move the cuts. It also costs one embedding call per sentence at indexing time. In a systematic comparison, Qu et al. (2024) found that its computational cost was “not justified by consistent performance gains” over fixed-size chunking. Chroma’s evaluation of chunking strategies likewise found that the choice of strategy matters (up to 9% in recall) and that heuristic splitters like the recursive one “often perform well in practice when parametrized appropriately”.

Better ideas: give each chunk its context

The deeper problem is that a chunk is embedded alone. “Reset links expire after 30 minutes” doesn’t say which product’s reset links. Two techniques attack that directly:

• Contextual retrieval (Anthropic, 2024): before embedding, have an LLM write a sentence or two situating each chunk in its document (“This chunk is from the security handbook’s section on password resets…”) and prepend it. Anthropic measured top-20 retrieval failures dropping by 35% with contextual embeddings, 49% when also applied to BM25, and 67% with a reranker on top.
• Late chunking (Günther et al., 2024): run a long-context embedding model over the whole document first, then pool the token vectors of each chunk’s span separately. Every chunk vector is computed with the full document in view, at no extra model calls.

Practical defaults

And always measure: build 30 to 50 real questions with their answer passages, track recall@k, and change one setting at a time.

References