Embeddings and search

An embedding is a vector, a list of numbers, that stands for a text's meaning: texts that mean much the same have vectors close together. You store a record's embedding in a field and find the records nearest a question with a query. PostgreSQL holds and searches them through its pgvector extension.

Storing a vector

A field of a vector type with dimensions: is stored in a pgvector vector(n) column. The type also says how its vectors are compared, since that depends on the model that made them:

/// A help article's meaning, as the configured embedding model makes it.type ArticleMeaning = Vector {    dimensions: 1536    distance: VectorDistance.cosine}

use support/types { ArticleMeaning, ArticleSlug, ArticleTitle }/// An article on the help site, with its meaning once it is embedded.entity HelpArticle {    id: Uuid    slug: ArticleSlug {        unique: true    }    title: ArticleTitle    body: Markdown    shown_publicly: Bool    body_meaning: ArticleMeaning?}

Its migration creates the extension first, since this is the store's first vector column, and the column takes the type's dimensions:

-- store: mainCREATE EXTENSION IF NOT EXISTS "vector";CREATE TABLE "help_article" (    "id" uuid NOT NULL,    "slug" text NOT NULL,    "title" text NOT NULL,    "body" text NOT NULL,    "shown_publicly" boolean NOT NULL,    "body_meaning" vector(1536),    CONSTRAINT "help_article_pkey" PRIMARY KEY ("id"));

Use cosine unless your model's documentation says otherwise. Smaller is always nearer, so inner_product gives pgvector's negative inner product as it is.

distance: fits any type built on Vector, with or without dimensions:. A type built on another keeps its metric, and giving it a different one is type.constraint_contradiction. With no distance: anywhere in the chain of types, it is cosine.

A plain Vector, with no dimensions:, is stored as vector with no length and can't be searched (query.nearest_unindexed). An Embedding is stored in three columns: its model's provider and name, as a ModelId is, and its vector in a vector without a length, since one column would lose which model made it. A vector column checks no length beyond what its type says: vector(n) keeps the dimensions, and pgvector keeps its limit of 1 to 16,000 items. A vector goes to PostgreSQL in pgvector's binary format, sent with no type number, so the server infers vector from the column it meets. That is because the extension's type number differs in each database. A column of it is known by its type's name.

ai.distance works out the same distance in code. It follows pgvector's C source: each item a single-precision float, the sums kept in single precision item by item, then the square root or the division in double precision. The result is the shortest decimal that is that double, as PostgreSQL writes a float8 (rounded to a decimal's 38 places below 1e-21). Where the items' products and running sums are exact floats, the two agree to the bit. Elsewhere they can differ in a float's last place, since pgvector's compiler vectorises and may add up in an order that depends on the server (within 1e-4, relative, on model-like vectors). A negative zero, the inner product of vectors at right angles, is 0. A sum past a float's range, which pgvector answers with infinity, is a decimal overflow.

ai is a standard namespace, so a domain, library or variable called ai is semantic.standard_namespace_reserved, as with every standard namespace (Words that are names everywhere else). Two vectors of different types can't be compared (type.mismatch), even with the same dimensions and metric, since they may come from different models. Nor can a vector that may be none:

/// A help article's meaning, as the configured embedding model makes it.type ArticleMeaning = Vector {    dimensions: 1536}/// Whether two articles say much the same, by the angle between them.type MuchTheSame = Bool {}function much_the_same(a: ArticleMeaning, b: ArticleMeaning) -> MuchTheSame {    return MuchTheSame(ai.distance(a, b) <= 0.2)}

A vector compared by cosine needs a direction, so a vector of zeros breaks distance, as a vector of the wrong length breaks dimensions. pgvector answers a cosine from it with NaN, and its index leaves it out, so a search would treat it differently from a scan. It is refused wherever it is made, so no row Jadpo writes holds one:

• Read from JSON, it is a mismatch.
• Built from a list only the running program knows, the construction's failure for distance: must be mapped.
• Written out in code, it is type.invalid_literal. A list written out passes when one item is clear of zero (above 2e-30 either way, since an item below 1e-30 is held as 0).

A vector compared by straight-line distance or inner product may be all zeros. Here a model's reply is read as a vector compared by cosine:

type ModelReply = Text {}type MeaningUnreadable = Failure {    kind: FailureKind.invalid_value}/// A help article's meaning, compared by cosine, the default.type ArticleMeaning = Vector {    dimensions: 3}/// A help article's meaning from the model's reply: JSON that isn't/// one, a vector of another length, or one with no direction.function article_meaning(model_reply: ModelReply) fails MeaningUnreadable -> ArticleMeaning {    return attempt json.read(model_reply, as: ArticleMeaning) {        malformed: MeaningUnreadable        mismatch: MeaningUnreadable    }}

test "a meaning has a direction" {    var read_back: ArticleMeaning = attempt article_meaning("[0, 0, 0.5]")    assert read_back == ArticleMeaning([0.0, 0.0, 0.5])    assert article_meaning("[0, 0, 0]") fails MeaningUnreadable    assert article_meaning("[0.5, 0, 0.5, 0]") fails MeaningUnreadable}

/// A help article's meaning, compared by cosine, the default.type ArticleMeaning = Vector {    dimensions: 3}/// A meaning to start a search from before anything is known.function blank_meaning() -> ArticleMeaning {    return ArticleMeaning([0.0, 0.0, 0.0])}

error: `[0.0,0.0,0.0]` isn't a valid `ArticleMeaning` --> main.jadpo:8:27  |8 |     return ArticleMeaning([0.0, 0.0, 0.0])  |                           ^^^^^^^^^^^^^^^ breaks `distance: VectorDistance.cosine`  reason: `Vector` declares `distance: VectorDistance.cosine`, and the value is a vector of zeros, which has no direction to compare by cosine. A literal is checked against every rule while compiling.  next: Use a value that `ArticleMeaning` allows (agent)  rule: type.invalid_literal  help: https://jadpo.dev/docs/errors/type.invalid_literalcheck failed: 1 error, 1 file

The nearest-neighbour query

To find the records whose vectors are nearest one you have, a query orders by nearness with nearest. Its result is a list of Nearest<T>, a standard type holding item: T and distance: Decimal, as Page<T> holds a page. Here the query finds the five public help articles nearest a question:

use support/entities { HelpArticle }/// What a reader sees of a help article in a list of answers.type ArticleExcerpt = Output {    pick: [HelpArticle.id, HelpArticle.slug, HelpArticle.title]}

use support/entities { HelpArticle }use support/types { ArticleExcerpt, ArticleMeaning }/// The five public articles nearest a question, none further than 0.6.query closest_articles(question_meaning: ArticleMeaning) -> List<Nearest<ArticleExcerpt>> {    from: HelpArticle    where: HelpArticle.shown_publicly == true    where: ai.distance(HelpArticle.body_meaning, question_meaning) <= 0.6    order: [HelpArticle.body_meaning nearest question_meaning]    limit: 5}

It runs as three settings and one statement, which jadpo sql closest_articles prints:

-- support.closest_articlesSET LOCAL hnsw.ef_search = 40;SET LOCAL hnsw.iterative_scan = strict_order;SET LOCAL hnsw.max_scan_tuples = 2147483647;SELECT *FROM (  SELECT "id", "slug", "title", "body_meaning" <=> $1 AS "jadpo_distance"  FROM "help_article"  WHERE "shown_publicly" = $2    AND "body_meaning" <=> $1 <= $3    AND "body_meaning" IS NOT NULL  ORDER BY "body_meaning" <=> $1  LIMIT 5) AS "nearest"ORDER BY "jadpo_distance", "id";

• nearest is an order: entry, and the only one in its query. It goes on a vector field with dimensions:; on any other vector field, an Embedding field included since its vector has no fixed length, it is query.nearest_unindexed. It measures from a parameter of the field's own type, or a field of one, never a vector written out (query.nearest_type otherwise). A value to measure from that may be none is query.optional_value.

• The result is List<Nearest<T>>, and the query needs a limit: of at most 100. A limit: above 100, or a value whose type allows more, is query.limit_range. A field that isn't a vector, a result that isn't List<Nearest<T>>, and a Nearest<T> result without nearest are query.nearest_type. A Page<T> of nearest records is refused, since a cursor over distances isn't stable as rows change (query.nearest_page). Records whose vector is none are left out.

• The order. The inner statement orders by distance alone, which is what the index serves. PostgreSQL won't read an HNSW index for an order with a second key, so the tie-break on id is applied outside, to the rows the index found. (With , id in the same ORDER BY, the plan is a scan of the whole table and a sort.)

• distance is the distance pgvector worked out: its float8 read by its bits and written as ai.distance writes the same double. Where the sums are exact, it is the same to the bit as ai.distance in code. A cosine that pgvector answers with NaN, from a vector of zeros written outside Jadpo, is a fault, never a made-up distance.

• ai.distance(a, b) in a query. ai.distance is a standard operation on two vectors of one type, giving that type's distance. In code it is worked out by the program. In a query's where:, whether the query orders by nearest or not, it becomes pgvector's operator. There you compare it with <, <=, > or >=, the field first and a parameter of its type second (type.mismatch otherwise, as in code), against a number written out or a decimal or whole-number parameter. == and != are query.condition_unsupported, since two computations of a float may differ in its last place. So is a distance in where_present:.

• What never matches. A record whose vector is none meets no distance condition, not included. Nor does one whose distance is NaN or an infinity, which code never gives, so a row whose vector was written outside Jadpo meets nothing. PostgreSQL sorts NaN above infinity and infinity above every number. So >, >= and the not of < and <= each add distance < 'Infinity'. An inner product can be minus infinity, so it adds distance > '-Infinity' to < and <=.

• Code and the database agree on the bound. Code compares ai.distance's decimal, the shortest that is the double, with the bound. PostgreSQL would compare the double with its own rounding of the bound, which differs for a bound of 17 or more places. So the bound is sent as a float8: the largest double whose decimal is at most the bound (or below it, for < and >=). Every comparison becomes "at most" or "above" that double, and the two agree on every double. A bound the caller gives is worked out as the read runs.

• Filters still fill the limit. An approximate index filters after it scans, so with a where: it can return fewer rows than limit:. To prevent that, the runtime:

  • sets hnsw.ef_search to the larger of pgvector's default, 40, and the most records the query returns (its limit:, or the max: of its value's type);
  • turns on pgvector's iterative scans (0.8.0 and later) in strict order;
  • sets hnsw.max_scan_tuples to pgvector's most, 2,147,483,647. pgvector's default, 20,000, would end a filtered search early, silently.

  A scan then ends only when it has filled the limit or visited every record, so a filtered query fills its limit when enough rows match. The memory a scan may use stays pgvector's bound, work_mem times hnsw.scan_mem_multiplier, at the server's settings. Raising it is tuning for the database, and a scan that reaches it still ends early.

• Nearest<T> stays out of routes. A route returns only an Output type, or a list or page of one (Output and status), so code maps the items it needs.