query.nearest_exact: Nearest-neighbour order on a vector too long for an index reads every record

Nearest-neighbour order on a vector too long for an index reads every record.

Area 21, AI; declinable; the next step is usually the agent's.

nearest on a vector type of over 4,000 dimensions. pgvector's HNSW index holds at most 4,000, at half precision, so the query reads and measures every record on each call. It is declinable: for a table small enough to read in full, a decline above the query's order: accepts the cost, and the query searches exactly.

example

/// A paper's meaning, as the research team's own model makes it.type PaperMeaning = Vector {    dimensions: 5000}/// A research paper, with its meaning once it is embedded.entity Paper {    id: Uuid    abstract_meaning: PaperMeaning?}/// The ten papers nearest a question.query closest_papers(question_meaning: PaperMeaning) -> List<Nearest<Paper>> {    from: Paper    order: [Paper.abstract_meaning nearest question_meaning]    limit: 10}

text

error: `Paper.abstract_meaning` is too long to index, so `nearest` reads every `Paper` --> main.jadpo:15:13   |15 |     order: [Paper.abstract_meaning nearest question_meaning]   |             ^^^^^^^^^^^^^^^^^^^^^^ 5,000 dimensions  reason: pgvector's HNSW index takes a vector of up to 4,000 dimensions, at half precision, and `PaperMeaning` has 5,000; with no index, each call measures the distance to every `Paper` whose `abstract_meaning` is set and sorts them all, which slows as the table grows.  next: Give `PaperMeaning` at most 4,000 `dimensions:`, asking the model for fewer with `embedding_dimensions:`, or write `decline query.nearest_exact: "<why>"` above it if this is deliberate (agent)  rule: query.nearest_exact  help: https://jadpo.dev/docs/errors/query.nearest_exactcheck failed: 1 error, 1 file

Declined, it checks, and runs as one statement:

example

/// A paper's meaning, as the research team's own model makes it.type PaperMeaning = Vector {    dimensions: 5000}/// A research paper, with its meaning once it is embedded.entity Paper {    id: Uuid    abstract_meaning: PaperMeaning?}/// The ten papers nearest a question.query closest_papers(question_meaning: PaperMeaning) -> List<Nearest<Paper>> {    from: Paper    decline query.nearest_exact: "Under a thousand papers, read in full in a few milliseconds"    order: [Paper.abstract_meaning nearest question_meaning]    limit: 10}

sql

SELECT "id", "abstract_meaning", "abstract_meaning" <=> $1 AS "jadpo_distance"FROM "paper"WHERE "abstract_meaning" IS NOT NULLORDER BY "jadpo_distance", "id"LIMIT 10